creativecommons / creativecommons/quantifying

Implement Internet Archive Data Fetching Pipeline

Abierto
#196 8 comentarios 0 reacciones 0 asignados Ver en GitHub
✨ goal: improvement 🏁 status: ready for work 💻 aspect: code help wanted 🟩 priority: low
Lenguaje dominante
Python
Estrellas
48
Forks
74
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Problem

We currently lack a mechanism for systematically and regularly harvesting license-aware metadata from the Internet Archive (IA), a massive source of CC-licensed content. This data is essential for understanding the large-scale adoption and usage of Creative Commons licenses on the platform.

## Proposal / Solution
Implement a new Python script and pipeline specifically designed to query the Internet Archive's API, process the results, and generate aggregated statistics.

## Scope of Work (What this pipeline will do)
1. Query: Search the IA for all items with a Creative Commons/open-source license.
2. Data Fetching: Implement robust API calling with exponential backoff and retry logic to handle network and rate-limiting issues across large result sets.
3. Normalization: Clean and standardize the messy raw license URLs provided by the IA using a lookup table (ia_license_mapping.csv).
4. Aggregation: Collect counts for licenses, languages, and countries.
5. Output: Save the aggregated counts into three separate, version-controlled CSV files.
6. Automation: Include optional flags (--enable-save, --enable-git) to support dry runs and automated Git commits/pushes for regular data updates.

## Expected Outcome
We have a baseline and a recurring process for measuring CC usage on the Internet Archive.

## Related Links
- [Tools and APIs — Internet Archive Developer Portal](https://archive.org/developers/index-apis.html)
- [Internet Archive: Search Engine](https://archive.org/advancedsearch.php)
- [jjjake/internetarchive](https://github.com/jjjake/internetarchive): _A Python and Command-Line Interface to Archive.org_
- [Searching Items — Developer Interface — Internet Archive Developer Portal](https://archive.org/developers/internetarchive/api.html#searching-items)
- Pre-automation effort:
- [`pre-automation/internetarchive/`](https://github.com/creativecommons/quantifying/tree/925e7212e98ffe0ee166439c10ae951cf41304dc/pre-automation/internetarchive)
- [`pre-automation/sources.md`](https://github.com/creativecommons/quantifying/blob/925e7212e98ffe0ee166439c10ae951cf41304dc/pre-automation/sources.md)

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza leyendo pre-automation/internetarchive/ y pre-automation/sources.md; después, consulta la documentación enlazada de Internet Archive API e inspecciona ia_license_mapping.csv. Se considera terminado cuando una canalización Python recurrente consulta y normaliza los metadatos de licencias de IA, agrega los recuentos de licencias, idiomas y países en tres CSVs bajo control de versiones, y admite los flags --enable-save y --enable-git.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
data-engineering
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.