Allow user DAGs to emit DagWarning (or similar)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
I would like a way to emit a warning or error message from a DAG that shows up as a banner in the web UI, similar to how exceptions are handled during DAG parsing or how [`DagWarning`](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dagwarning/index.html)s are displayed.
### Use case/motivation
I have a script under my `dags_folder` that is dynamically discovering and building DAGs from a networked filesystem. If one dynamically discovered DAG raises an exception, I don't want that to affect any of the other DAGs that are being discovered, so I swallow exceptions and log about them. However, I have not found a good way to communicate to my users that that exception occurred. Ideally I think it should look similar to when an exception is raised in a traditional DAG - an error banner on the web with (optionally) the full traceback.
In my testing, I've found that if *any* exception is raised during DAG parsing, it causes *all* DAGs originating from that file to be removed. So this sort of thing does not work:
```
#!/usr/bin/env python3
from datetime import datetime
import traceback
from airflow.decorators import dag, task
exceptions = []
for dag_id in 'ABCDE':
try:
@dag(
dag_id=dag_id,
schedule=None,
start_date=datetime(2023, 1, 31),
)
def test_multi_exception():
if dag_id in 'BE':
raise RuntimeError(f'Failed to create dynamic DAG {dag_id}')
@task
def t():
print('inside task')
t()
test_multi_exception()
except Exception as e:
exceptions.append(''.join(traceback.format_exception(etype=type(e), value=e, tb=e.__traceback__)))
if exceptions:
raise RuntimeError('\n'.join(exceptions))
```
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with the DagWarning documentation and the DAG parsing behavior described in the issue, including the dags_folder discovery example. Define how a user DAG emits a warning or error that appears as a web UI banner, optionally with a traceback, without removing unrelated DAGs from the same file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100