internetarchive / internetarchive/warcprox

concurrency bug when running with multiple warc writer threads

Open
#101 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
467
Forks
66
PR merge metrics
No merged PRs in 30d

Description

In July @vbanos reported invalid gzip data in a warc written by warcprox with `--writer-threads=5`.

My benchmarking suggests that 1 writer thread is optimal:
https://github.com/internetarchive/warcprox/wiki/benchmarking-number-of-threads
At IA we are running warcprox with 1 writer thread everywhere.

It would be nice to find and fix this bug, but the pragmatic thing might be to remove support for multiple writer threads.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the benchmarking results linked in the issue and reproduce the invalid gzip data with warcprox using --writer-threads=5, comparing it with one writer thread. Done means either the concurrent writer path no longer produces invalid WARC gzip data or support for multiple writer threads is removed, but the issue does not name files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.