High process count+fast CPU -> deadlock

[klauspost]

On my 16 core CPU I very often have the fuzzer lock up. Contrary to #262 this happens without fuzzers crashing, but seems highly correlated to process count.

Lockups happen in 9 out of 10 times when starting the fuzzer. Applying #276 does not fix the issue. Running a CPU consuming task when starting the fuzzer makes it much more likely to "succeed".

go get -u github.com/minio/simdjson-fuzz
go get -u github.com/minio/simdjson-go
cd $GOPATH$/src/github.com/minio/simdjson-fuzz
go-fuzz-build -o=fuzz-build.zip -func=Fuzz .
go-fuzz -bin=fuzz-build.zip -workdir=corpus -procs=32

This will stop executing, either right away or after a minute or 2.

Example

go-fuzz -minimize=5s -bin=fuzz-build.zip -workdir=corpus -procs=32

2020/02/24 11:06:10 workers: 32, corpus: 7631 (1s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s
2020/02/24 11:06:13 workers: 32, corpus: 7631 (4s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 2220, uptime: 6s
2020/02/24 11:06:16 workers: 32, corpus: 7631 (7s ago), crashers: 0, restarts: 1/5253, execs: 63039 (6558/sec), cover: 2220, uptime: 9s
2020/02/24 11:06:19 workers: 32, corpus: 7631 (10s ago), crashers: 0, restarts: 1/5933, execs: 71204 (5646/sec), cover: 2220, uptime: 12s
2020/02/24 11:06:22 workers: 32, corpus: 7631 (13s ago), crashers: 0, restarts: 1/5933, execs: 71204 (4561/sec), cover: 2220, uptime: 15s
2020/02/24 11:06:25 workers: 32, corpus: 7631 (16s ago), crashers: 0, restarts: 1/5933, execs: 71204 (3825/sec), cover: 2220, uptime: 18s
2020/02/24 11:06:28 workers: 32, corpus: 7631 (19s ago), crashers: 0, restarts: 1/5933, execs: 71204 (3295/sec), cover: 2220, uptime: 21s
2020/02/24 11:06:31 workers: 32, corpus: 7631 (22s ago), crashers: 0, restarts: 1/5933, execs: 71204 (2893/sec), cover: 2220, uptime: 24s

Example 2

λ go-fuzz -bin=fuzz-build.zip -workdir=corpus -procs=32
2020/02/24 11:18:44 workers: 32, corpus: 7641 (2s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s
2020/02/24 11:18:47 workers: 32, corpus: 7641 (5s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 2207, uptime: 6s
2020/02/24 11:18:50 workers: 32, corpus: 7641 (8s ago), crashers: 0, restarts: 1/3659, execs: 146371 (15217/sec), cover: 2220, uptime: 9s
2020/02/24 11:18:53 workers: 32, corpus: 7641 (11s ago), crashers: 0, restarts: 1/4989, execs: 234483 (18582/sec), cover: 2220, uptime: 12s
2020/02/24 11:18:56 workers: 32, corpus: 7641 (14s ago), crashers: 0, restarts: 1/5554, execs: 311064 (19917/sec), cover: 2220, uptime: 15s
2020/02/24 11:18:59 workers: 32, corpus: 7641 (17s ago), crashers: 0, restarts: 1/6261, execs: 406966 (21857/sec), cover: 2220, uptime: 18s
2020/02/24 11:19:02 workers: 32, corpus: 7641 (20s ago), crashers: 0, restarts: 1/6741, execs: 498879 (23076/sec), cover: 2220, uptime: 21s
2020/02/24 11:19:05 workers: 32, corpus: 7641 (23s ago), crashers: 0, restarts: 1/7038, execs: 591227 (24016/sec), cover: 2220, uptime: 24s
2020/02/24 11:19:08 workers: 32, corpus: 7641 (26s ago), crashers: 0, restarts: 1/7298, execs: 678755 (24576/sec), cover: 2220, uptime: 27s
2020/02/24 11:19:11 workers: 32, corpus: 7641 (29s ago), crashers: 0, restarts: 1/7625, execs: 762586 (24906/sec), cover: 2220, uptime: 30s
2020/02/24 11:19:14 workers: 32, corpus: 7641 (32s ago), crashers: 0, restarts: 1/7769, execs: 846889 (25191/sec), cover: 2220, uptime: 33s
2020/02/24 11:19:17 workers: 32, corpus: 7641 (35s ago), crashers: 0, restarts: 1/7914, execs: 933955 (25505/sec), cover: 2220, uptime: 36s
2020/02/24 11:19:20 workers: 32, corpus: 7641 (38s ago), crashers: 0, restarts: 1/8052, execs: 1022718 (25814/sec), cover: 2220, uptime: 39s
2020/02/24 11:19:23 workers: 32, corpus: 7641 (41s ago), crashers: 0, restarts: 1/8205, execs: 1107753 (25992/sec), cover: 2220, uptime: 42s
2020/02/24 11:19:26 workers: 32, corpus: 7641 (44s ago), crashers: 0, restarts: 1/8277, execs: 1191977 (26129/sec), cover: 2220, uptime: 45s
2020/02/24 11:19:29 workers: 32, corpus: 7641 (47s ago), crashers: 0, restarts: 1/8383, execs: 1274355 (26211/sec), cover: 2220, uptime: 48s
2020/02/24 11:19:32 workers: 32, corpus: 7641 (50s ago), crashers: 0, restarts: 1/8450, execs: 1360450 (26356/sec), cover: 2220, uptime: 51s
2020/02/24 11:19:35 workers: 32, corpus: 7641 (53s ago), crashers: 0, restarts: 1/8557, execs: 1446290 (26479/sec), cover: 2220, uptime: 54s
2020/02/24 11:19:38 workers: 32, corpus: 7641 (56s ago), crashers: 0, restarts: 1/8670, execs: 1534619 (26634/sec), cover: 2220, uptime: 57s
2020/02/24 11:19:41 workers: 32, corpus: 7641 (59s ago), crashers: 0, restarts: 1/8670, execs: 1621372 (26747/sec), cover: 2220, uptime: 1m0s
2020/02/24 11:19:44 workers: 32, corpus: 7641 (1m2s ago), crashers: 0, restarts: 1/8739, execs: 1677956 (26375/sec), cover: 2220, uptime: 1m3s
2020/02/24 11:19:47 workers: 32, corpus: 7641 (1m5s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (26117/sec), cover: 2220, uptime: 1m6s
2020/02/24 11:19:50 workers: 32, corpus: 7641 (1m8s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (24992/sec), cover: 2220, uptime: 1m9s
2020/02/24 11:19:53 workers: 32, corpus: 7641 (1m11s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (23959/sec), cover: 2220, uptime: 1m12s
2020/02/24 11:19:56 workers: 32, corpus: 7641 (1m14s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (23009/sec), cover: 2220, uptime: 1m15s
2020/02/24 11:19:59 workers: 32, corpus: 7641 (1m17s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (22130/sec), cover: 2220, uptime: 1m18s
2020/02/24 11:20:02 workers: 32, corpus: 7641 (1m20s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (21317/sec), cover: 2220, uptime: 1m21s
2020/02/24 11:20:05 workers: 32, corpus: 7641 (1m23s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (20561/sec), cover: 2220, uptime: 1m24s
2020/02/24 11:20:08 workers: 32, corpus: 7641 (1m26s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (19857/sec), cover: 2220, uptime: 1m27s
2020/02/24 11:20:11 workers: 32, corpus: 7641 (1m29s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (19200/sec), cover: 2220, uptime: 1m30s
2020/02/24 11:20:14 workers: 32, corpus: 7641 (1m32s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (18585/sec), cover: 2220, uptime: 1m33s
2020/02/24 11:20:17 workers: 32, corpus: 7641 (1m35s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (18007/sec), cover: 2220, uptime: 1m36s
2020/02/24 11:20:20 workers: 32, corpus: 7641 (1m38s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (17465/sec), cover: 2220, uptime: 1m39s
2020/02/24 11:20:23 workers: 32, corpus: 7641 (1m41s ago), crashers: 0, restarts: 1/8743, execs: 1739874 (16955/sec), cover: 2220, uptime: 1m42s

-v=5 output: v5-debug-output.txt.gz

Using -procs=8 or something similar low makes the deadlock very unlikely. So it seems to be related to how fast fuzz executions are.

There are several Fuzz functions Fuzz (used here) FuzzCorrect, FuzzSerialize. The deadlock happens with all, and also on other fuzzers.

[klauspost]

(investigating)