creativecommons / creativecommons/quantifying

Make Europeana fetch viable

Open
#224 5 comments 0 reactions 0 assignees View on GitHub
✨ goal: improvement 💻 aspect: code 🔒 staff only 🚧 status: blocked 🟩 priority: low
Dominant language
Python
Stars
48
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Problem
The Europeana data set is too big to fetch all at once, or in its entirety.
- data providers: 4206
- rights: 64

Assuming approximately 1 second per query, the expected duration is approximately 75 hours.

## Description
1. Add counts for rights only (not including data providers)
2. Write a query plan on first run per quarter to map out which queries to make over 30 runs (30 days)

## Alternatives
Parallel queries (probably considered impolite, Timid Robot will ask)

## Additional context
- https://github.com/creativecommons/quantifying/blob/main/sources.md#europeana

## Implementation

- [ ] I would be interested in implementing this feature.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.