Split testing and example dags in loading
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
As described in https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
After #52469 and #52474 are implemented - another increment is to split example and testing dags which are today somehow interlinked. The following rule should apply:
- pure example dags are stored per provider in /src/airflow/providers//example_dags/
- Testing dags should be moved to /src/airflow/providers//testing_dags/
- Dual use Dags which are examples but also used for testing would be stored with example_dags but tagged for testing
The dynamic dag bundle (or whatever mechanism) that collects examples would also need to be extended to source testing dags where provided per provider.
We can automatically add such dags to be used in unit test via auto-fixture in the shared conftest.py or similar - this way it will work in both breeze and local venv.
Also on breeze cli the testing dags should be loadable similar like the CLI command `--load-example-dags` via `--load-testing-dags`. Same liek a Airflow Conf entry should be available so that a standard deployment can use it.
### Use case/motivation
Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:
Serve as tutorials to learn Airflow DAG implementation
Serve with code snippets for documentation
Serve for testing the setup
(some) service for CI integration testing
Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.
There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.
So in the [Airflow 3 Dev Calls](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025) there was a demand named to clean-up and optimize example DAGs.
### Related issues
#52469 and #52474 as pre-requisite
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading the linked Examples Refurbish proposal and prerequisite issues #52469 and #52474. Trace how the dynamic DAG bundle, shared conftest.py, and Breeze's --load-example-dags path currently collect and load provider example_dags. Done means testing_dags are separated, discoverable for unit tests, loadable through Breeze, and covered by an Airflow configuration entry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100