internetarchive / internetarchive/warc
.gz WARC files not properly read
- Dominant language
- Python
- Stars
- 250
- Forks
- 114
- PR merge metrics
- No merged PRs in 30d
Description
When reading WARC files compressed with gzip, many of the entries contained are skipped or misread. To reproduce, use common crawl data in .gz format, count the number of entries found by the WARC library and then count the number of appearances of WARC/1.0 in the file. It is a very large difference.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with a Common Crawl .gz WARC file. Compare the number of entries returned by the WARC library with the number of WARC/1.0 occurrences, then trace the gzip-reading path to determine where entries are skipped or misread. Done means compressed WARC files yield the expected entries without loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100