DAG-level `on_failure_callback` never fires
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
3.1.7
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
DAG-level `on_failure_callback` does not fire in Airflow 3.1.7 with KubernetesExecutor.
Task-level `on_failure_callback` (via `default_args`) works correctly — it fires on the worker pod.
### Observations
1. **DagRun reaches `failed` state** — confirmed via `SELECT state FROM dag_run`. The `last_scheduling_decision` and `end_date` are within milliseconds of each other.
2. **`has_on_failure_callback` is present in serialized data** — confirmed via `SELECT data::text LIKE '%has_on_failure_callback%' FROM serialized_dag`.
3. **Scheduler DEBUG logs show `"callback is empty"` for every DagRun** — this is the log from `_send_dag_callbacks_to_processor` when `callback_to_run` is `None`, meaning `update_state()` returned no callback.
4. **DAG processor logs show no callback-related activity** — only routine DAG file parsing (`Setting next_dagrun ... to None`).
### What you think should happen instead?
When `on_failure_callback` is set on a DAG and the DagRun fails, the callback should fire. This worked in Airflow 2.x.
### How to reproduce
1. Create a DAG with `on_failure_callback` set at the DAG level:
```python
from datetime import datetime
from airflow.sdk import DAG
from airflow.providers.standard.operators.python import PythonOperator
def my_callback(context):
print("DAG failure callback fired")
def task_that_fails():
raise ValueError("Intentional error")
with DAG(
dag_id="test-dag-level-callback",
schedule=None,
start_date=datetime(2025, 1, 1),
catchup=False,
on_failure_callback=[my_callback],
) as dag:
PythonOperator(
task_id="task_that_fails",
python_callable=task_that_fails,
)
```
2. Trigger the DAG. The task fails, the DagRun is marked as `failed`.
3. The DAG-level `on_failure_callback` never fires. No output in scheduler logs, DAG processor logs, or task logs.
4. Enable DEBUG logging on the scheduler (`AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG`) and search for `"callback is empty"` — it appears for every DagRun.
5. For comparison, move the callback to `default_args`:
```python
with DAG(
dag_id="test-task-level-callback",
schedule=None,
start_date=datetime(2025, 1, 1),
catchup=False,
default_args={"on_failure_callback": [my_callback]},
) as dag:
...
```
This works — the task-level callback fires on the worker pod.
### Operating System
Linux
### Versions of Apache Airflow Providers
```
apache-airflow 3.1.7
apache-airflow-core 3.1.7
apache-airflow-providers-amazon 9.21.0
apache-airflow-providers-celery 3.15.2
apache-airflow-providers-cncf-kubernetes 10.12.3
apache-airflow-providers-common-compat 1.13.0
apache-airflow-providers-common-io 1.7.1
apache-airflow-providers-common-messaging 2.0.2
apache-airflow-providers-common-sql 1.30.4
apache-airflow-providers-docker 4.5.2
apache-airflow-providers-elasticsearch 6.4.4
apache-airflow-providers-fab 3.2.0
apache-airflow-providers-ftp 3.14.1
apache-airflow-providers-git 0.2.2
apache-airflow-providers-google 19.5.0
apache-airflow-providers-grpc 3.9.2
apache-airflow-providers-hashicorp 4.4.3
apache-airflow-providers-http 5.6.4
apache-airflow-providers-microsoft-azure 12.10.3
apache-airflow-providers-mysql 6.4.2
apache-airflow-providers-odbc 4.11.1
apache-airflow-providers-openlineage 2.10.1
apache-airflow-providers-postgres 6.5.3
apache-airflow-providers-redis 4.4.2
apache-airflow-providers-sendgrid 4.2.1
apache-airflow-providers-sftp 5.7.0
apache-airflow-providers-slack 9.6.2
apache-airflow-providers-smtp 2.4.2
apache-airflow-providers-snowflake 6.9.0
apache-airflow-providers-ssh 4.3.1
apache-airflow-providers-standard 1.11.0
apache-airflow-task-sdk 1.1.7
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.