creativecommons / creativecommons/quantifying

[Feature] Automate Flickr Data Fetching

Offen
#164 12 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
✨ goal: improvement 💻 aspect: code 🚧 status: blocked 🟩 priority: low
Vorherrschende Sprache
Python
Sterne
48
Forks
74
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

> [!CAUTION]
> Because it requires a paid subscription (Flickr Pro), Flickr is not currently a viable data source.

## Problem
I've been exploring the codebase and noticed that Flickr data collection is still in `pre-automation/flickr/` while other sources like Google Custom Search and GitHub have been automated. The current Flickr scripts appear to be manual and haven't been integrated with the automated pipeline yet.

## Description
Looking at `pre-automation/flickr/photos.py`, it currently:
- Searches for photos with different Creative Commons licenses (IDs: 1,2,3,4,5,6,9,10)
- Gets 500 photos per license
- Saves everything to a JSON file

I noticed some differences compared to the automated scripts:
- **Output**: Flickr uses JSON, but GCS/GitHub use CSV
- **Arguments**: The automated scripts have `--enable-save`, `--enable-git` etc.
- **Setup**: Automated scripts use `shared.setup()` instead of the old `quantify.setup()`
- **Integration**: They integrate with git automation and quarterly folders

The `Pipfile` already mentions `flickrapi` and has a planned script path: `flickr_fetched = "./scripts/1-fetch/flickr_fetched.py"`

## Implementation Plan

*Following the guidance to plan each phase before implementation, and after studying the 2025Q3 report structure:*

### Research Findings
The GCS report is way more detailed than I expected, it's not just counting licenses, but showing breakdowns by different countries, languages, and grouping licenses into categories. All the charts look the same with bars on the left and pie charts on the right, and they use the same colors throughout.

Looking at this made me understand why planning carefully is important, with such a large Flickr dataset, I could easily collect tons of data that doesn't actually help with making useful reports.

### Fetch Phase
**Build new automation (using pre-automation as reference only):**
- Create `scripts/1-fetch/flickr_fetch.py` following the same pattern as `gcs_fetch.py`
- Use the same setup method, save to CSV files, same command options
- Focus on collecting data that will actually be useful for the reporting phase
- Get license types (1,2,3,4,5,6,9,10) plus any metadata that supports analysis

### Process Phase
**Transform data to support reporting:**
- Map Flickr license IDs to actual CC license names and versions using the [Flickr API reference](https://www.flickr.com/services/api/flickr.photos.licenses.getInfo.html)
- Group licenses the same way as GCS (Latest/Prior/Retired categories)
- Process any geographic or language data to match the report structure
- Make sure everything aligns with what will actually be shown in charts

### Report Phase
**Create visualizations that add value:**
- License distribution analysis (similar to GCS "Products totals")
- Status breakdown (Latest/Prior/Retired)
- Geographic or language analysis if the data supports it
- Focus on analyses that make sense for visual content

### Next Steps
I'd like to start by understanding the license mapping and what kinds of reports would be most useful. Then build the fetch script to collect only the data that will actually be used in reporting.

This way I can avoid doing a lot of fetch work that won't end up being helpful for the final reports.

## Alternatives
- Build Flickr automation following a different pattern than GCS/GitHub
- Focus on other data sources first and return to Flickr later
- Start with a smaller subset of licenses to test the approach

## Additional Context
- I'm still learning the codebase patterns, but I can see the structure used by other automated sources
- The scripts use Creative Commons license IDs that Flickr recognizes
- Currently limited to 500 photos per license and outputs to JSON format
- Could potentially be modernized to follow the pattern used by other automated sources in `scripts/1-fetch/`
- Pre-automation effort:
- [`pre-automation/flickr/`](https://github.com/creativecommons/quantifying/tree/925e7212e98ffe0ee166439c10ae951cf41304dc/pre-automation/flickr)
- [`pre-automation/sources.md`](https://github.com/creativecommons/quantifying/blob/925e7212e98ffe0ee166439c10ae951cf41304dc/pre-automation/sources.md)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginnen Sie mit der Lektüre von pre-automation/flickr/photos.py, scripts/1-fetch/gcs_fetch.py, des Pipfile-Eintrags für flickrapi und der Struktur des 2025Q3-Berichts. Klären Sie zunächst, ob der Zugriff auf Flickr Pro diese Arbeit erlaubt; die im Issue vorgeschlagenen Phasen für Abruf, Verarbeitung und Berichterstellung sind weit gefasst, und die Abschlusskriterien sind noch nicht definiert.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
analytics, data-engineering
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.