apache / apache/airflow

Split testing and example dags in loading

Open
#52,475 2 comments 2 reactions 0 assignees View on GitHub
area:core kind:feature kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

As described in https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish

After #52469 and #52474 are implemented - another increment is to split example and testing dags which are today somehow interlinked. The following rule should apply:
- pure example dags are stored per provider in /src/airflow/providers//example_dags/
- Testing dags should be moved to /src/airflow/providers//testing_dags/
- Dual use Dags which are examples but also used for testing would be stored with example_dags but tagged for testing

The dynamic dag bundle (or whatever mechanism) that collects examples would also need to be extended to source testing dags where provided per provider.

We can automatically add such dags to be used in unit test via auto-fixture in the shared conftest.py or similar - this way it will work in both breeze and local venv.
Also on breeze cli the testing dags should be loadable similar like the CLI command `--load-example-dags` via `--load-testing-dags`. Same liek a Airflow Conf entry should be available so that a standard deployment can use it.

### Use case/motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

Serve as tutorials to learn Airflow DAG implementation
Serve with code snippets for documentation
Serve for testing the setup
(some) service for CI integration testing

Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.

There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.

So in the [Airflow 3 Dev Calls](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025) there was a demand named to clean-up and optimize example DAGs.

### Related issues

#52469 and #52474 as pre-requisite

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Examples Refurbish proposal and prerequisite issues #52469 and #52474. Trace how the dynamic DAG bundle, shared conftest.py, and Breeze's --load-example-dags path currently collect and load provider example_dags. Done means testing_dags are separated, discoverable for unit tests, loadable through Breeze, and covered by an Airflow configuration entry.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.