apache / apache/airflow

Add mechanism for DAG files to report errors without raising exception/loosing all DAGs in a file

Open
#53,971 10 comments 3 reactions 0 assignees View on GitHub
area:core area:DAG-processing kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

With dynamic DAGs (either "old school" of `for dag_id in some_list` or using something like Dag Factory) it would be nice to be able to report an error to show up in the Airflow UI without loosing all the other dags in the file.

Right now this is not possible as the only way of reporting an DAG error is to raise an exception, but if the exception is raise Airflow doesn't look for any DAGs in the loaded dag file/module anymore.

We should add _some_ mechanism that lets these tools or dynamic dags report a list of exceptions so that they can still parse/load the valid dags but have a more natural experience for reporting errors.

I don't have any concrete ideas on this, but it will likely have to be done by setting some "magic"/special variable at the module level that contains a list of exceptions. (The Dag Parsing process already iterates over all the attributes on the module looking for dags, so looking for this special var, say `__airflow_dag_parsing_errors`, wouldn't be a big lift.

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing Airflow's DAG parsing process and how parser errors reach the UI, then determine where valid DAG discovery stops after an exception. Done means a documented mechanism can report multiple DAG-file errors while still loading valid DAGs, with coverage for both behaviors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.