apache / apache/airflow

Automated checks where possible for example dags

Open
#52,477 1 comment 1 reaction 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

As described in https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish

For all checks that are possible to be automated (e.g. enforcing proper pydoc per example dag) checks are implemented.

This ticket might be splitted and does not define a fixed list but candidates might be:
- Check for pydoc on module as well as function/task level
- Check that display names are attached to all dags and tasks
- Check that doc_md is provided
- Check that tasks are properly typed (via mypy strict mode and ruff?)
- Check that tag "example" is added for all examples, tag according to provider and if testing then the tag "testing"
- No deprecated functions being used (via ruff?)

### Use case/motivation

Airflow Examples have been grown in number and focus over the past years. They purpose multiple things:

Serve as tutorials to learn Airflow DAG implementation
Serve with code snippets for documentation
Serve for testing the setup
(some) service for CI integration testing

Some example DAGs are in a good quality, some are not following best practices. Current examples do not follow a structure.

There are example DAGs contained in the Airflow core (currently pushed to standard provider/example_dags) as well as there are more examples in other providers. But examples from other providers are lot loaded automatically.

So in the [Airflow 3 Dev Calls](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025) there was a demand named to clean-up and optimize example DAGs.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the Examples Refurbish proposal and inspect the example DAGs in standard provider/example_dags. Identify which candidate checks are feasible and agree on a focused scope before implementing them. Done means the selected automated checks are enforced for the relevant example DAGs and pass against the existing examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, data-engineering, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.