creativecommons / creativecommons/quantifying

Automate data pipeline orchestration using Apache Airflow

Abierto
#182 4 comentarios 0 reacciones 0 asignados Ver en GitHub
✨ goal: improvement 💬 talk: discussion 💻 aspect: code 🚧 status: blocked 🟩 priority: low
Lenguaje dominante
Python
Estrellas
48
Forks
74
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

## Problem
Each data source in the project currently follows a similar workflow that involves fetching, processing, and report generation. However, these steps are executed through individual scripts, which can make the process difficult to automate, monitor, and maintain — especially when scaling to multiple data sources.

## Description
I propose using Apache Airflow to orchestrate and automate the data workflows.
Each data source can be represented as an Airflow DAG, with tasks corresponding to each stage:

- Fetch: Collect data from APIs or external sources.

- Process: Clean, transform, and aggregate the data.

- Report: Generate summaries or metrics for analysis.

This approach would improve visibility, scheduling, error handling, and reusability across all data pipelines.

## Alternatives

- Continue using manual script execution

- Implement lightweight scheduling within Python scripts.

## Additional context
This integration aligns with the existing three-phase workflow structure and can easily wrap around current scripts without major codebase refactoring. Airflow also supports modular task development, which would benefit future contributors.

## Implementation

- [x] I would be interested in implementing this feature.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

No se nombran archivos, pruebas ni puntos de entrada específicos. Empieza por localizar los scripts existentes de fetch, process y report para cada fuente de datos y, a continuación, determina cómo podrían asignarse a tareas de Airflow DAG; se considerará terminado cuando los workflows estén programados, sean observables y admitan el manejo de errores sin alterar las fases existentes.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
data-engineering
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.