Corpus collection times out
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
@chenju2k6 @jonathanmetzman @Alan32Liu
In #1827 the experiment reports only `NaN` for a variety of different fuzzers, with a specific bias towards `Symsan` and my tool.
I was able to reproduce these issues locally by making a set of test fuzzers to test out suspicions I had that could be causing it.
The root cause for both of the tools in this test seem to be the large logs they print to `stdout`/`stderr`. Unlike the `corpus` archives, the `results` are collected and copied/rsynced around uncompressed which causes dramatic slow downs.
However, during my testing I identified another cause that could be problematic for some fuzzers: large log-files with `append` semantics. I have a few of these in my fuzzer where I continuously append the current tick's information.
Unfortunately, because of the way `fuzzbench` performs the differential backups on a file-by-file basis (if the file changed, copy it in full again), this also can take a long time and gets slower and slower as time goes on. This unfortunately prevents the simple solution of just appending a `2>/out/stderr 1>/out/corpus/stdout` to the offending fuzzers, because then the archiving will still have the same issues. At least in this case it gets compressed before being copied around, however.
In the way of a solution, I have deployed a `multilog` based log scheme now where the logs are rotated to new files once they exceed a certain capacity, which can make use of the by-file-differential copying better and keeps each file of manageable size.
I enabled this for `symsan` and my tool now, would you want to consider including it upstream? The script is rather simple and would in that case only add a `daemontools` dependency to the `base-image` and copy in a `run_with_multilog.sh` script that fuzzers can use by doing `/run_with_multilog.sh $OUT/.log/ ` for example.
**TL;DR: fuzzbench handles large logs and large append-only files very badly and it affects symsan's and my results. It should either be added to the README to make sure it's known that this causes problems, or maybe a solution like my run_with_multilog.sh solution can be deployed in the infrastructure itself**
Lastly, I think for future insights it might be good to include logging for time taken for the measurement, corpus copying, report generation etc. so such bottlenecks can be found. At the moment the only way to see it is that the wait period in `runner-log-xxxx.txt` in the `dispatcher-container` starts going negative. Maybe a prometheus + grafana solution to see time taken for each component for the current experiment or something like that so this can be caught early while the experiment is running.
Contributor guide
Assessment
This issue has not been assessed yet.