Add mechanism for DAG files to report errors without raising exception/loosing all DAGs in a file
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
With dynamic DAGs (either "old school" of `for dag_id in some_list` or using something like Dag Factory) it would be nice to be able to report an error to show up in the Airflow UI without loosing all the other dags in the file.
Right now this is not possible as the only way of reporting an DAG error is to raise an exception, but if the exception is raise Airflow doesn't look for any DAGs in the loaded dag file/module anymore.
We should add _some_ mechanism that lets these tools or dynamic dags report a list of exceptions so that they can still parse/load the valid dags but have a more natural experience for reporting errors.
I don't have any concrete ideas on this, but it will likely have to be done by setting some "magic"/special variable at the module level that contains a list of exceptions. (The Dag Parsing process already iterates over all the attributes on the module looking for dags, so looking for this special var, say `__airflow_dag_parsing_errors`, wouldn't be a big lift.
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
No files or tests are named. Start by tracing Airflow's DAG parsing process and how parser errors reach the UI, then determine where valid DAG discovery stops after an exception. Done means a documented mechanism can report multiple DAG-file errors while still loading valid DAGs, with coverage for both behaviors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100