creativecommons / creativecommons/quantifying
Make Europeana fetch viable
- Dominant language
- Python
- Stars
- 48
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The Europeana data set is too big to fetch all at once, or in its entirety.
- data providers: 4206
- rights: 64
Assuming approximately 1 second per query, the expected duration is approximately 75 hours.
## Description
1. Add counts for rights only (not including data providers)
2. Write a query plan on first run per quarter to map out which queries to make over 30 runs (30 days)
## Alternatives
Parallel queries (probably considered impolite, Timid Robot will ask)
## Additional context
- https://github.com/creativecommons/quantifying/blob/main/sources.md#europeana
## Implementation
- [ ] I would be interested in implementing this feature.
Contributor guide
Assessment
This issue has not been assessed yet.