Task-level failure callbacks are not triggered when tasks are SIGKILLed
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
3.1.6
### If "Other Airflow 3 version" selected, which one?
3.1.2
### What happened?
When a task is OOM-killed (container/process terminated with SIGKILL), the task appears to fail silently:
Task-level on_failure_callback is not executed
In logs/behavior, the executor reports the task as SUCCESS
Because the scheduler never observes a FAILED state (and no mismatch is detected), it does not trigger task-level failure handling/callbacks
### What you think should happen instead?
When a task process is terminated with SIGKILL:
The executor should report the task as FAILED (or otherwise indicate abnormal termination)
Airflow should then trigger task-level on_failure_callback (and any related failure handling)
### How to reproduce
Create a task which can OOM and has task level failure call back and it wont be triggered
```from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
def func():
a = "asd"
while True:
a += a*100000
def task_failure_callback(context):
"""Callback function called when a task fails."""
task_instance = context.get('task_instance')
dag_run = context.get('dag_run')
print(f"Task {task_instance.task_id} failed in DAG {dag_run.dag_id} with run_id {dag_run.run_id}")
with DAG(
dag_id="oom_example",
start_date=datetime(2024, 1, 1),
schedule="@daily",
catchup=False,
doc_md=__doc__,
tags=["oom"],
) as dag:
oom_task = PythonOperator(
task_id="oom_task",
python_callable=func,
retries=0,
on_failure_callback=task_failure_callback,
)
```
### Operating System
debian
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Assessment
This issue has not been assessed yet.