internetarchive / internetarchive/warctools

record dumper assumes content type and content length

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
178
Forks
33
PR merge metrics
No merged PRs in 30d

Description

As stated in [http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/#content-type]

> All records with a non-empty block (non-zero Content-Length), except ‘continuation’ records, **should** have a Content-Type field. Only if the media type is not given by a Content-Type field, a reader may attempt to guess the media type via inspection of its content and/or the name extension(s) of the URI used to identify the resource. If the media type remains unknown, the reader should treat it as type “application/octet-stream”.

This is a should not a must. The [record dumper](https://github.com/internetarchive/warctools/blob/master/hanzo/warctools/record.py#L150) should not assume that a record has a content type or content length. It currently crashes on such records but should be able to handle such cases.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.