internetarchive / internetarchive/warcprox
concurrency bug when running with multiple warc writer threads
- Dominant language
- Python
- Stars
- 467
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
In July @vbanos reported invalid gzip data in a warc written by warcprox with `--writer-threads=5`.
My benchmarking suggests that 1 writer thread is optimal:
https://github.com/internetarchive/warcprox/wiki/benchmarking-number-of-threads
At IA we are running warcprox with 1 writer thread everywhere.
It would be nice to find and fix this bug, but the pragmatic thing might be to remove support for multiple writer threads.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the benchmarking results linked in the issue and reproduce the invalid gzip data with warcprox using --writer-threads=5, comparing it with one writer thread. Done means either the concurrent writer path no longer produces invalid WARC gzip data or support for multiple writer threads is removed, but the issue does not name files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100