Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parser implementation #139

Merged
merged 20 commits into from
Aug 29, 2023
Merged

Parser implementation #139

merged 20 commits into from
Aug 29, 2023

Conversation

teo-tsirpanis
Copy link
Owner

No description provided.

@teo-tsirpanis
Add classes representing parser messages and lexical errors.
c6c5584
@teo-tsirpanis
Start writing the default tokenizer.
a7b98b6
@teo-tsirpanis
Add an optimized stack type.
ddfcc37 
This is a port of Farkle 6's `StackNeo`.
@teo-tsirpanis
Add internal grammar APIs related to groups.
95037d7
@teo-tsirpanis
Support tokenizing groups.
9d2a0a9
@teo-tsirpanis
Update documentation.
1be2ba8
@teo-tsirpanis
Add the default parser implementation.
4d768c6
@teo-tsirpanis
Format and localize syntax errors.
50479b2
@teo-tsirpanis
Add APIs to create CharParsers.
0b34d83
@teo-tsirpanis
Rename ParserCommon to ParserUtilities.
d13f5c4
@teo-tsirpanis
Fix bugs and SonarCloud suggestions.
bc3a169
@teo-tsirpanis
Do not discard the last token of a non-final block if the DFA cannot …
0540aad 
…proceed from its current state.
@teo-tsirpanis
Wrap the result of Tokenizer.Create in a chain.
4349edf
@teo-tsirpanis
Fix a nullability mismatch.
4780068
@teo-tsirpanis
Add some tests for the tokenizer.
7f0fbd6
@teo-tsirpanis
Fix bugs and oversights in the grammar code.
dd69b7b
@teo-tsirpanis
Track in ParserState if suspending the tokenizer is supported.
55b8450 
I thought that suspending the tokenizer without being in a chain would have no effect, but it actually has the effect of prohibiting subsequent suspensions.
With this change, suspending is a real no-op if the tokenizer is not wrapped in a chain.
@teo-tsirpanis
Add a benchmark comparing parsing JSON in Farkle 6 and 7.
5983a99
@teo-tsirpanis
Fix a bug in CharacterBufferManager.
3958c81 
`GetWriteBufferOffset` can resize the buffer, but calls to `_buffer.AsSpan(GetWriteBufferOffset(sizeHint))` always use the old buffer.
The code was restructured to make the correct way to do it more apparent.
@teo-tsirpanis
Push the initial state to the LR parser stack.
112e674 
Turns out it is always needed to do GOTOs from the initial state if we reduce a production with as many members as the stack.
And with that we don't need to separately track the LR state anymore; it's at the top of the stack.
@sonarcloud
Copy link

sonarcloud bot commented Aug 29, 2023 •
edited
Loading

SonarCloud Quality Gate failed.    Quality Gate failed

Bug E 1 Bug
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 4 Code Smells

16.7% 16.7% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@teo-tsirpanis
Copy link
Owner Author

Initial benchmarks


BenchmarkDotNet v0.13.7, Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Intel Xeon Platinum 8272CL CPU 2.60GHz, 1 CPU, 2 logical and 2 physical cores
.NET SDK 7.0.400
  [Host]     : .NET 7.0.10 (7.0.1023.36312), X64 RyuJIT AVX2
  DefaultJob : .NET 7.0.10 (7.0.1023.36312), X64 RyuJIT AVX2
Method FileName Mean Error StdDev Ratio Gen0 Allocated Alloc Ratio
Farkle6String big.json 1,318.409 μs 2.3033 μs 2.1545 μs 1.00 17.5781 338794 B 1.000
Farkle7String big.json 3,934.867 μs 1.8726 μs 1.6600 μs 2.98 - 545 B 0.002
Farkle6String medium.json 71.060 μs 0.1325 μs 0.1239 μs 1.00 0.9766 19816 B 1.00
Farkle7String medium.json 204.647 μs 0.8334 μs 0.7795 μs 2.88 - 536 B 0.03
Farkle6String small.json 8.611 μs 0.0114 μs 0.0089 μs 1.00 0.1984 3912 B 1.00
Farkle7String small.json 16.201 μs 0.0142 μs 0.0126 μs 1.88 - 536 B 0.14
Farkle6Stream big.json 1,412.930 μs 2.0664 μs 1.9329 μs 1.00 17.5781 342154 B 1.00
Farkle7Stream big.json 4,284.979 μs 6.2011 μs 5.8005 μs 3.03 - 4561 B 0.01
Farkle6Stream medium.json 80.108 μs 0.1033 μs 0.0916 μs 1.00 1.2207 23176 B 1.00
Farkle7Stream medium.json 211.409 μs 0.1998 μs 0.1771 μs 2.64 - 4552 B 0.20
Farkle6Stream small.json 10.024 μs 0.0204 μs 0.0180 μs 1.00 0.3815 7272 B 1.00
Farkle7Stream small.json 17.206 μs 0.0800 μs 0.0748 μs 1.72 0.2136 4552 B 0.63

🥲

@teo-tsirpanis teo-tsirpanis marked this pull request as ready for review August 29, 2023 22:09
@teo-tsirpanis
Copy link
Owner Author

Let's merge it right now. The parser will be tested when we port the F# tests in a subsequent PR, and benchmarks already caught some bugs. Performance optimizations will happen in a subsequent PR.

@teo-tsirpanis teo-tsirpanis merged commit 564f200 into mainstream Aug 29, 2023
6 of 7 checks passed
@teo-tsirpanis teo-tsirpanis deleted the parser-impl branch August 29, 2023 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@teo-tsirpanis