Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuzz tests are not discovered by test runner when re-running with same --seed argument #21410

Open
AdamGoertz opened this issue Sep 14, 2024 · 2 comments
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. fuzzing
Milestone

Comments

@AdamGoertz
Copy link
Contributor

Zig Version

0.14.0-dev.1573+4d81e8ee9

Steps to Reproduce and Observed Behavior

When running zig build test --fuzz --seed <seed> multiple times, on the second and later runs, the test runner exits with an error

$ ../zig/build/release/bin/zig build test --fuzz --seed 100
info: web interface listening at http://127.0.0.1:38091/
info: hint: pass --port 38091 to use this same port next time
^C/1] Fuzzing
$ ../zig/build/release/bin/zig build test --fuzz --seed 100
error: no fuzz tests found
error: the following build command failed with exit code 1:
/home/adam/fuzz_test/.zig-cache/o/8faf9687350f461d281e202daf8c0a3b/build /home/adam/zig/build/release/bin/zig /home/adam/zig/lib /home/adam/fuzz_test /home/adam/fuzz_test/.zig-cache /home/adam/.cache/zig --seed 100 -Z9b313e175a6f1d65 test --fuzz

This does not occur when a different seed is used, or when --seed is not set.

Different seeds:

$ ../zig/build/release/bin/zig build test --fuzz --seed 102
info: web interface listening at http://127.0.0.1:32903/
info: hint: pass --port 32903 to use this same port next time
^C/1] Fuzzing
$ ../zig/build/release/bin/zig build test --fuzz --seed 103
info: web interface listening at http://127.0.0.1:45863/
info: hint: pass --port 45863 to use this same port next time
^C/1] Fuzzing

No --seed

$ ../zig/build/release/bin/zig build test --fuzz
info: web interface listening at http://127.0.0.1:45375/
info: hint: pass --port 45375 to use this same port next time
^C/1] Fuzzing
$ ../zig/build/release/bin/zig build test --fuzz
info: web interface listening at http://127.0.0.1:43537/
info: hint: pass --port 43537 to use this same port next time
^C/1] Fuzzing

Expected Behavior

Fuzz tests should run even when the same seed is used multiple times.

@AdamGoertz AdamGoertz added the bug Observed behavior contradicts documented or intended behavior label Sep 14, 2024
@Vexu Vexu added the fuzzing label Sep 14, 2024
@Vexu Vexu added this to the 0.15.0 milestone Sep 14, 2024
@andrewrk
Copy link
Member

@Vexu thanks for helping triage. Can you let me pick the milestone for stuff that I'm working on? I'm doing a big fuzzing push this release cycle, so I would have liked to see this issue when filtering for "no:milestone".

@andrewrk andrewrk modified the milestones: 0.15.0, 0.14.0 Sep 14, 2024
@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. and removed bug Observed behavior contradicts documented or intended behavior labels Sep 14, 2024
@andrewrk
Copy link
Member

This is effectively a duplicate of #20803. I'll leave it open for now and address it along with that one. I have this partially completed locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Solving this issue will likely involve adding new logic or components to the codebase. fuzzing
Projects
None yet
Development

No branches or pull requests

3 participants