apache / apache/airflow

Allow user DAGs to emit DagWarning (or similar)

Open
#29,275 5 comments 2 reactions 0 assignees View on GitHub
good first issue kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

I would like a way to emit a warning or error message from a DAG that shows up as a banner in the web UI, similar to how exceptions are handled during DAG parsing or how [`DagWarning`](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dagwarning/index.html)s are displayed.

### Use case/motivation

I have a script under my `dags_folder` that is dynamically discovering and building DAGs from a networked filesystem. If one dynamically discovered DAG raises an exception, I don't want that to affect any of the other DAGs that are being discovered, so I swallow exceptions and log about them. However, I have not found a good way to communicate to my users that that exception occurred. Ideally I think it should look similar to when an exception is raised in a traditional DAG - an error banner on the web with (optionally) the full traceback.

In my testing, I've found that if *any* exception is raised during DAG parsing, it causes *all* DAGs originating from that file to be removed. So this sort of thing does not work:
```
#!/usr/bin/env python3
from datetime import datetime
import traceback

from airflow.decorators import dag, task

exceptions = []
for dag_id in 'ABCDE':
try:
@dag(
dag_id=dag_id,
schedule=None,
start_date=datetime(2023, 1, 31),
)
def test_multi_exception():
if dag_id in 'BE':
raise RuntimeError(f'Failed to create dynamic DAG {dag_id}')

@task
def t():
print('inside task')

t()

test_multi_exception()

except Exception as e:
exceptions.append(''.join(traceback.format_exception(etype=type(e), value=e, tb=e.__traceback__)))

if exceptions:
raise RuntimeError('\n'.join(exceptions))
```

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the DagWarning documentation and the DAG parsing behavior described in the issue, including the dags_folder discovery example. Define how a user DAG emits a warning or error that appears as a web UI banner, optionally with a traceback, without removing unrelated DAGs from the same file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.