Update the tokenizer composability model.

[teo-tsirpanis]

While porting the samples to FarkleNeo, it was discovered that the existing tokenizer extensibility model was not adequate to implement the indent-based grammar. This PR makes the following changes:

• Instead of going through the chain once, the chained tokenizer keeps running all components until:
  • One of them returns true.
  • One of them suspends.
  • No input characters were consumed after all components had the opportunity to run and returned false.
• Tokenizer implementations are no longer greedy. When they encounter a noise symbol they must consume its characters and return false instead of continuing until they encounter a terminal.
  • Tokenizers can forgo yielding if they are the only ones in the chain, and an extension method on ParserInputReader<TChar> was added to check for it.
    • Checking for this property is mandatory for tokenizers that can skip the chained tokenizer wrapping.
  • The chained tokenizer wrapping maintains its eager semantics, which means that from the perspective of the parser, if a tokenizer returns false, the parser must still exit and return with more available input.

The new design was validated locally by successfully running the indent-based grammar checks.

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
84.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud