Add new standard metadata (optional) to differentiate content harvesting period from ZIM creation date
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 9
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Currently, there is only one standard metadata named Date in ZIM metadata. Documentation specifically states this is the ZIM creation date.
There is no standard metadata to store information about when the ZIM content has been captured / fetched / crawled / scraped / ...
Given the fact that we rebuild regularly ZIMs (see ZIM Update v2 at https://github.com/openzim/overview/issues/35 and https://wiki.openzim.org/wiki/ZIM_Updates) and we more and more process content that has been harvested at a time different than the ZIM creation (all stackexchange, some zimit with warcs reprocessed), it is useful to consider adding a new standard metadata to store this information.
Given the fact that content (e.g. with zimit) can be scrapped across multiple days, it seems important that the date is in fact a range from-to.
Just like current Date metadata, I think that we should keep this metadata understandable / easy to grab by keeping it only a day, not a day+time.
Given the fact that some content might come with lower precision than a day (e.g. when a content provider says "this is the content for April 2023, do not mind which day I published it"), I think we need to allow passing only a month or only a year in this metadata.
I hence propose to introduce this new standard ZIM metadata:
- Name:
ContentDate - Mandatory:
No - Description:
Date of the content, i.e. when content has been fetched to create the ZIM ; preferably a day (ISO format YYYY-MM-DD) but can be a year (YYYY) or month (YYYY-MM) if daily precision makes no sense ; can be a single value or a range from start to end, with format "from,to" - Examples:
2012-11or2023-01-12,2023-01-15
WDYT?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the ZIM Update v2 issue and the ZIM Updates wiki page linked in the discussion. Resolve whether ContentDate should be a single value or range, how reduced precision is represented, and whether the proposed format fits existing ZIM metadata conventions. Done means the standard is agreed and its name, optionality, description, and examples are documented.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100