creativecommons / creativecommons/quantifying
[Feature] Add Openverse as a Data Source
- Dominant language
- Python
- Stars
- 48
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The project currently has GitHub and GCS as automated data sources, but not Openverse. Openverse provides a large collection of openly licensed media, which will greatly enhance the breadth and depth of this data observatory
## Description
Openverse aggregates data from several other openly licensed repositories like Flickr. It provides:
- quantity of records: millions
- types of metadata available: `source`, `license`, `license_version`, `media_type` etc
- API documentation link: https://api.openverse.engineering/v1/, https://api.openverse.org/v1/images/
- API requirements and limitations:
- no API key required
- rate limits are low but sufficient for batch collection
- supports filtering by `license` and `media_type`
## Alternatives
- Work on another source
## Additional context
- Still understanding the project and solving this issue with one simple PR at a time
- Openverse is compatible with the project structure for tracking CC Legal tools usage
## Implementation
- I will be implementing this feature
- Focus on a single non-monolithic script `scripts/1-fetch/openverse_fetch.py`
- Design script to run from the repository via `pipenv`
- include `--enable-save` and `--enable-git` for consistent behavior with other scripts
Contributor guide
Assessment
This issue has not been assessed yet.