creativecommons / creativecommons/quantifying
[Feature] Add Openverse as a Data Source
- Lingua principale
- Python
- Stelle
- 48
- Fork
- 74
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Problem
The project currently has GitHub and GCS as automated data sources, but not Openverse. Openverse provides a large collection of openly licensed media, which will greatly enhance the breadth and depth of this data observatory
## Description
Openverse aggregates data from several other openly licensed repositories like Flickr. It provides:
- quantity of records: millions
- types of metadata available: `source`, `license`, `license_version`, `media_type` etc
- API documentation link: https://api.openverse.engineering/v1/, https://api.openverse.org/v1/images/
- API requirements and limitations:
- no API key required
- rate limits are low but sufficient for batch collection
- supports filtering by `license` and `media_type`
## Alternatives
- Work on another source
## Additional context
- Still understanding the project and solving this issue with one simple PR at a time
- Openverse is compatible with the project structure for tracking CC Legal tools usage
## Implementation
- I will be implementing this feature
- Focus on a single non-monolithic script `scripts/1-fetch/openverse_fetch.py`
- Design script to run from the repository via `pipenv`
- include `--enable-save` and `--enable-git` for consistent behavior with other scripts
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.