apache / apache/airflow

New example along storyline: Dynamic Dag generation

Open
#52,478 1 comment 1 reaction 1 assignee Claimed by @shubhamraj-git View on GitHub
area:dynamic-task-mapping kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

As described in https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish

A new Dag is implemented along the described storyline for "Dynamic dag generation" This includes:

- New Dag is added in Taskflow as well as classic implementation
- Dag is according to checklist
- Previous examples that are content-wise replaced are deleted to incrementally clean-up
- Airflow documentation is adjusted to point to new examples are replaced
- General example storyline overview is adjusted and the functioning of the new example is described

### Use case/motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

Serve as tutorials to learn Airflow DAG implementation
Serve with code snippets for documentation
Serve for testing the setup
(some) service for CI integration testing

Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.

There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.

So in the [Airflow 3 Dev Calls](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025) there was a demand named to clean-up and optimize example DAGs.

### Related issues

#52476 is made before and might be adjusted

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.