creativecommons / creativecommons/quantifying

Implement Internet Archive Data Fetching Pipeline

Open
#196 8 comments 0 reactions 0 assignees View on GitHub
✨ goal: improvement 🏁 status: ready for work 💻 aspect: code help wanted 🟩 priority: low
Dominant language
Python
Stars
48
Forks
74
PR merge metrics
No merged PRs in 30d

Description

## Problem

We currently lack a mechanism for systematically and regularly harvesting license-aware metadata from the Internet Archive (IA), a massive source of CC-licensed content. This data is essential for understanding the large-scale adoption and usage of Creative Commons licenses on the platform.

## Proposal / Solution
Implement a new Python script and pipeline specifically designed to query the Internet Archive's API, process the results, and generate aggregated statistics.

## Scope of Work (What this pipeline will do)
1. Query: Search the IA for all items with a Creative Commons/open-source license.
2. Data Fetching: Implement robust API calling with exponential backoff and retry logic to handle network and rate-limiting issues across large result sets.
3. Normalization: Clean and standardize the messy raw license URLs provided by the IA using a lookup table (ia_license_mapping.csv).
4. Aggregation: Collect counts for licenses, languages, and countries.
5. Output: Save the aggregated counts into three separate, version-controlled CSV files.
6. Automation: Include optional flags (--enable-save, --enable-git) to support dry runs and automated Git commits/pushes for regular data updates.

## Expected Outcome
We have a baseline and a recurring process for measuring CC usage on the Internet Archive.

## Related Links
- [Tools and APIs — Internet Archive Developer Portal](https://archive.org/developers/index-apis.html)
- [Internet Archive: Search Engine](https://archive.org/advancedsearch.php)
- [jjjake/internetarchive](https://github.com/jjjake/internetarchive): _A Python and Command-Line Interface to Archive.org_
- [Searching Items — Developer Interface — Internet Archive Developer Portal](https://archive.org/developers/internetarchive/api.html#searching-items)
- Pre-automation effort:
- [`pre-automation/internetarchive/`](https://github.com/creativecommons/quantifying/tree/925e7212e98ffe0ee166439c10ae951cf41304dc/pre-automation/internetarchive)
- [`pre-automation/sources.md`](https://github.com/creativecommons/quantifying/blob/925e7212e98ffe0ee166439c10ae951cf41304dc/pre-automation/sources.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.