internetarchive / internetarchive/warctools
record dumper assumes content type and content length
- Dominant language
- Python
- Stars
- 178
- Forks
- 33
- PR merge metrics
- No merged PRs in 30d
Description
As stated in [http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/#content-type]
> All records with a non-empty block (non-zero Content-Length), except ‘continuation’ records, **should** have a Content-Type field. Only if the media type is not given by a Content-Type field, a reader may attempt to guess the media type via inspection of its content and/or the name extension(s) of the URI used to identify the resource. If the media type remains unknown, the reader should treat it as type “application/octet-stream”.
This is a should not a must. The [record dumper](https://github.com/internetarchive/warctools/blob/master/hanzo/warctools/record.py#L150) should not assume that a record has a content type or content length. It currently crashes on such records but should be able to handle such cases.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.