internetarchive / internetarchive/warc

.gz WARC files not properly read

Open
#21 11 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
250
Forks
114
PR merge metrics
No merged PRs in 30d

Description

When reading WARC files compressed with gzip, many of the entries contained are skipped or misread. To reproduce, use common crawl data in .gz format, count the number of entries found by the WARC library and then count the number of appearances of WARC/1.0 in the file. It is a very large difference.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with a Common Crawl .gz WARC file. Compare the number of entries returned by the WARC library with the number of WARC/1.0 occurrences, then trace the gzip-reading path to determine where entries are skipped or misread. Done means compressed WARC files yield the expected entries without loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.