apache / apache/airflow

Improving Import Errors for Dynamically Generated DAGs

Open
#55,475 3 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

Improved error propagation for dynamic DAG files. I propose introducing an exception type that PUTs an import error via the Airflow REST API without causing the dynamic dag loop to exit. To do this, a new API route would be created that allows users to PUT a DAG parsing error. Then we would create an ImportErrorException class that could be used in a try except block in the dynamic DAG code which would propagate the error. The errors would be cleaned up automatically when the dynamic dag is parsed. We could also implement a DELETE endpoint for import errors as persistent errors would be resurfaced by the dag parsing loop.

### Use case/motivation

A common pattern in Airflow is using a single file to generate DAGs dynamically from config files. If a user creates an invalid config, there are two general outcome

1. The entire dynamic dag generation process is interrupted and an import error is surfaced for the dynamic dag module. This approach is not ideal because one bad config can disrupt the every dynamic DAG. This can affect shared testing environments significantly.
2. The user has implemented a try/except block in the dynamic dag file and the import error is suppressed, likely with error level logs being emitted in the DAG processor. This approach is suboptimal for data engineers because the logs are emitted by the dag processor instead of surfaced as import errors.

Instead of either of these approaches, specific configuration error should be surfaced as import errors. Users should be able to propagate their own, richer errors as ImportErrors

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing the REST API route, the proposed ImportErrorException, and the dynamic DAG parsing loop; done should include propagating user-defined configuration errors without exiting the loop and cleaning them up as parsing state changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.