internetarchive / internetarchive/warctools
G-Zip Content-Length
Open
- Dominant language
- Python
- Stars
- 178
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
Warctools uses the Content-Length field to determine the length of the body for validating and reading WARC files. Since the g-zipped bodies are no longer g-zipped in common-crawl WARC files, not the whole of g-zipped messages is being parsed.
#14 fixes this and allows proper parsing common-crawl WARC files.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.