apache / apache/airflow

dag_processing.import_errors metric gives wrong value

Open
#60,928 1 comment 0 reactions 0 assignees View on GitHub
area:core kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

Other Airflow 3 version (please specify below)

### If "Other Airflow 3 version" selected, which one?

3.0.6

### What happened?

We have alerts based on dag_processing.import_errors. Even though there is no error in airflow UI, the metric is updated with some value.

On further checking, we found that whenever callback runs in dag processor, the particular dag is marked as Error. During this time, if the import_error metric is emitted, the value is updated, even though the dag does not have any errors.

### What you think should happen instead?

The import error metric should be updated when callback runs for a dag (or) the dag parser should not mark the dag as errored when callback runs.

### How to reproduce

Run 5-10 dag with multiple tasks in parallel. Make sure all tasks have failure callbacks and celery worker. Kill the celery worker and wait for the callback to run in dag-processor. Check the import error metric.

### Operating System

Debian GNU/Linux 12 (bookworm)

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.7.0
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-fab==2.4.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-http==5.3.3
apache-airflow-providers-imap==3.9.2
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-sqlite==4.1.2
apache-airflow-providers-standard==1.6.0

### Deployment

Other

### Deployment details

We use a slightly modified version of official helm chart

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by tracing dag_processing.import_errors through the DAG processor when a failure callback runs, using the reported reproduction with parallel tasks and a killed Celery worker. Compare the callback path with the parser's DAG error state, then verify that the metric reflects only actual import errors or that callback processing no longer marks the DAG as errored.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.