creativecommons / creativecommons/quantifying

[Feature] Add Openverse as a Data Source

Aperta
#184 5 commenti 1 reazione 1 assegnatario Rivendicata da @Babi-B Vedi su GitHub
✨ goal: improvement 💻 aspect: code 🔒 staff only 🚧 status: blocked 🟩 priority: low
Lingua principale
Python
Stelle
48
Fork
74
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Problem
The project currently has GitHub and GCS as automated data sources, but not Openverse. Openverse provides a large collection of openly licensed media, which will greatly enhance the breadth and depth of this data observatory

## Description
Openverse aggregates data from several other openly licensed repositories like Flickr. It provides:
- quantity of records: millions
- types of metadata available: `source`, `license`, `license_version`, `media_type` etc
- API documentation link: https://api.openverse.engineering/v1/, https://api.openverse.org/v1/images/
- API requirements and limitations:
- no API key required
- rate limits are low but sufficient for batch collection
- supports filtering by `license` and `media_type`

## Alternatives
- Work on another source

## Additional context
- Still understanding the project and solving this issue with one simple PR at a time
- Openverse is compatible with the project structure for tracking CC Legal tools usage

## Implementation
- I will be implementing this feature
- Focus on a single non-monolithic script `scripts/1-fetch/openverse_fetch.py`
- Design script to run from the repository via `pipenv`
- include `--enable-save` and `--enable-git` for consistent behavior with other scripts

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.