creativecommons / creativecommons/quantifying

[Feature] Add Openverse as a Data Source

Open
#184 5 comments 1 reaction 1 assignee Claimed by @Babi-B View on GitHub
✨ goal: improvement 💻 aspect: code 🔒 staff only 🚧 status: blocked 🟩 priority: low
Dominant language
Python
Stars
48
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Problem
The project currently has GitHub and GCS as automated data sources, but not Openverse. Openverse provides a large collection of openly licensed media, which will greatly enhance the breadth and depth of this data observatory

## Description
Openverse aggregates data from several other openly licensed repositories like Flickr. It provides:
- quantity of records: millions
- types of metadata available: `source`, `license`, `license_version`, `media_type` etc
- API documentation link: https://api.openverse.engineering/v1/, https://api.openverse.org/v1/images/
- API requirements and limitations:
- no API key required
- rate limits are low but sufficient for batch collection
- supports filtering by `license` and `media_type`

## Alternatives
- Work on another source

## Additional context
- Still understanding the project and solving this issue with one simple PR at a time
- Openverse is compatible with the project structure for tracking CC Legal tools usage

## Implementation
- I will be implementing this feature
- Focus on a single non-monolithic script `scripts/1-fetch/openverse_fetch.py`
- Design script to run from the repository via `pipenv`
- include `--enable-save` and `--enable-git` for consistent behavior with other scripts

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.