Use a unified index size for state machines.

[teo-tsirpanis]

State machines contain indices to many different things (states, edges, actions, tables…). For maximum size efficiency each of these indices can be compressed to a size of one, two or four bytes.

The problem with each index having its own size is that it results in thousands of generic type instantiations, because we specialize the state machine classes based on the type of each index. This results in huge NativeAOT binaries.

This PR changes the specification to have a single index size per state machine. It will be less size-efficient -if a DFA has say 100 states and 1000 edges in total, now indices to both states and edges will have a size of two bytes-, but the number of instantiations will be small. For comparison a sample console app that converts GOLD Parser grammars weighs 20MB before this change, after this it drops to 3.

Let's not merge it immediately, I want to also investigate forgoing generic descendants of classes, in favor of switching on the index size every time, as we do with grammar tables.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
86.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[teo-tsirpanis]

gml.grammar.dat went from 2.861 bytes to 3.919. 🤔

[teo-tsirpanis]

Benchmark report

GrammarReaderBenchmark

BenchmarkDotNet v0.14.0, Ubuntu 22.04.5 LTS (Jammy Jellyfish)
AMD EPYC 7763, 1 CPU, 4 logical and 2 physical cores
.NET SDK 9.0.100-rc.2.24474.11
  [Host]     : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2
  DefaultJob : .NET 8.0.10 (8.0.1024.46610), X64 RyuJIT AVX2

Method	Grammars	Mean	Error	StdDev	Ratio	Gen0	Gen1	Gen2	Allocated	Alloc Ratio
ConvertFarkle6	COBOL85	15,178,325.6 ns	140,915.38 ns	124,917.82 ns	1.00	93.7500	78.1250	-	8912505 B	1.00
ConvertFarkle7	COBOL85	7,514,005.8 ns	105,725.83 ns	93,723.19 ns	0.50	593.7500	585.9375	562.5000	5089651 B	0.57
ConvertFarkle6	JSON	85,136.4 ns	532.97 ns	498.54 ns	1.00	1.2207	-	-	110128 B	1.00
ConvertFarkle7	JSON	50,635.3 ns	98.35 ns	87.19 ns	0.59	0.3662	-	-	35560 B	0.32
ReadFarkle6	COBOL85	13,773,996.6 ns	150,425.57 ns	140,708.17 ns	1.000	78.1250	62.5000	-	6727625 B	1.000
ReadFarkle7	COBOL85	291,625.2 ns	942.93 ns	882.02 ns	0.021	-	-	-	360 B	0.000
ReadFarkle7NoValidation	COBOL85	206.0 ns	1.23 ns	1.15 ns	0.000	0.0043	-	-	360 B	0.000
ReadFarkle6	JSON	37,561.3 ns	208.72 ns	195.24 ns	1.000	0.4883	-	-	42728 B	1.000
ReadFarkle7	JSON	1,456.0 ns	12.04 ns	11.26 ns	0.039	0.0038	-	-	360 B	0.008
ReadFarkle7NoValidation	JSON	206.7 ns	0.91 ns	0.85 ns	0.006	0.0043	-	-	360 B	0.008

[teo-tsirpanis]

Superseded by #305. The performance differences are tiny (also ran the JSON benchmarks), but the grammar file size regression was significant, and the existing format is better by itself.