Reduce CI fuzz iterations as we're now timing out #3691
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I've assigned @joostjager as a reviewer! |
tnull
previously approved these changes
Mar 31, 2025
joostjager
reviewed
Mar 31, 2025
| else | ||
| HFUZZ_RUN_ARGS="$HFUZZ_RUN_ARGS -N1000000" | ||
| HFUZZ_RUN_ARGS="$HFUZZ_RUN_ARGS -N500000" |
There was a problem hiding this comment.
Any specific reasoning behind the 10x, 10x and 2x reductions?
There was a problem hiding this comment.
No. I probably could have benchmarked, but in general there's not a lot of value in full_stack_target in CI because its just too dense to make any real progress, and similar for chanmon_consistency_target.
|
I checked out a fuzzer run on the attr failures PR and grepped the logs. Interestingly it seems like indeed
Maybe I am not interpreting correctly though. Also from this log, you'd say that all the fast tests (<30 sec) don't need their iteration count changed. |
TheBlueMatt
force-pushed
the
2025-03-fuzz-less
branch
from
0244aa0 to
9f68f1b
Compare
Ah, thanks for doing that. Yea, I was going on the results when I fuzz with real corpuses, where
Huh, interesting, I went ahead and slowed this one down though.
Yep! |
TheBlueMatt
force-pushed
the
2025-03-fuzz-less
branch
2 times, most recently
from
c87c3f3 to
ad4fb1e
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3691 +/- ##
==========================================
- Coverage 89.18% 89.16% -0.02%
==========================================
Files 155 155
Lines 120796 120796
Branches 120796 120796
==========================================
- Hits 107731 107710 -21
- Misses 10415 10432 +17
- Partials 2650 2654 +4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
TheBlueMatt
force-pushed
the
2025-03-fuzz-less
branch
5 times, most recently
from
2773431 to
b93b1d6
Compare
When we made `test_node_counter_consistency` more aggressively run, our `process_network_graph` fuzzer got materially slower, resulting in consistent fuzz CI job timeouts. Thus, here, we tweak the iteration count on all our fuzz jobs to get them running in more consistent times. Further, we further reduce `full_stack_target` iterations in anticipation of a later commit which will start using our hard-coded fuzz seeds, creating substantially more coverage and slowing down fuzzing iterations.
In 3145168 we disabled `test_node_counter_consistency` in debug builds since it can make make things very slow, including `lightning-rapid-gossip-sync` tests. We should, however, have kept it when fuzzing, since that gives us testing of potential coverage gaps in normal tests.
This should materially improve our fuzzing coverage in CI.
TheBlueMatt
force-pushed
the
2025-03-fuzz-less
branch
from
b93b1d6 to
4b4bad8
Compare
joostjager
reviewed
Apr 4, 2025
|
Yea, happy to wait until we can at least run a fuzzing run and check that all the timings make sense. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I think github has slowed down the runners so now our fuzz tests are timing out. Here we just reduce iteration count a bit.