creativecommons / creativecommons/quantifying
Make Europeana fetch viable
- Lingua principale
- Python
- Stelle
- 48
- Fork
- 74
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
## Problem
The Europeana data set is too big to fetch all at once, or in its entirety.
- data providers: 4206
- rights: 64
Assuming approximately 1 second per query, the expected duration is approximately 75 hours.
## Description
1. Add counts for rights only (not including data providers)
2. Write a query plan on first run per quarter to map out which queries to make over 30 runs (30 days)
## Alternatives
Parallel queries (probably considered impolite, Timid Robot will ask)
## Additional context
- https://github.com/creativecommons/quantifying/blob/main/sources.md#europeana
## Implementation
- [ ] I would be interested in implementing this feature.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Start with the Europeana section linked from sources.md and trace the existing Europeana fetch entry point and query-counting flow. Define how rights-only counts and a first-run quarterly plan should cover 30 runs over 30 days, then verify that the resulting schedule avoids fetching the full dataset at once.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- data-engineering
- Tipo di issue
- Funzionalità
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 42/100