apache / apache/airflow

KubernetesPodOperator task is incorrectly marked as SUCCESS instead of FAILED when both the task pod and its child pod are interrupted in close succession

Open
#71,202 2 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

3.3.0

### What happened and how to reproduce it?

**Architecture context**

This happens when using `KubernetesExecutor` together with `KubernetesPodOperator`, which involves two separate pods:

1. **The per-task pod** created by `KubernetesExecutor` to execute the task instance.
2. **The child pod** created by `KubernetesPodOperator.execute()` to run the user's workload.

**Issue Description**

When both pods are interrupted within roughly one second of each other, the task instance is sometimes finalized as `success`, even though the child pod never completed its work.

**Steps to reproduce**

1. Trigger a DAG task using `KubernetesPodOperator` whose child pod runs long enough to still be running when interrupted. Example:
```
from datetime import datetime
from airflow import DAG
from airflow.models import Variable
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

default_args = {
"owner": "data",
"depends_on_past": False,
"start_date": datetime.strptime("2026-08-05", "%Y-%m-%d"),
"params": {
"severity": 0,
},
"retries": 1,
}

dag = DAG(
dag_id="air-test-sigterm-repro",
default_args=default_args,
catchup=False,
schedule=None,
max_active_runs=1
)

sleep_worker = KubernetesPodOperator(
dag=dag,
task_id="sleep-worker",
name="sleep-worker",
namespace=Variable.get("var-k8s-namespace"),
image="python:3.9",
cmds=["python", "-c"],
do_xcom_push=False,
get_logs=True,
is_delete_operator_pod=True,
arguments=[
"import time\n"
"for i in range(120):\n"
" print('tick', i, 'still running, still streaming logs', flush=True)\n"
" time.sleep(5)\n"
"print('finished without interruption', flush=True)\n"
],
)

```
2. While the task is running, delete **both** the child pod and the per-task pod within the same short time window (rather than waiting for one deletion to complete). The most reliable sequence we found is to delete the child pod first, then the per-task pod about one second later.

Terminal 1 — **child pod**:
```bash
kubectl delete pod sleep-worker-xxxxxxx -n airflow --grace-period=30
```

Terminal 2 (~1 second later) — **per-task pod**:
```bash
kubectl delete pod air-test-sigterm-repro-sleep-worker-xxxxxxxxx -n airflow --grace-period=30
```

3. Observe the task's final state in the Airflow UI or API.

Image

logs sleep-worker-xxxxxxx
```text
tick 0 still running, still streaming logs
tick 1 still running, still streaming logs
tick 2 still running, still streaming logs
tick 3 still running, still streaming logs
tick 4 still running, still streaming logs
tick 5 still running, still streaming logs
tick 6 still running, still streaming logs
tick 7 still running, still streaming logs
tick 8 still running, still streaming logs
tick 9 still running, still streaming logs
tick 10 still running, still streaming logs
tick 11 still running, still streaming logs
tick 12 still running, still streaming logs
tick 13 still running, still streaming logs
tick 14 still running, still streaming logs
tick 15 still running, still streaming logs
tick 16 still running, still streaming logs
tick 17 still running, still streaming logs
```
logs air-test-sigterm-repro-sleep-worker-xxxxxxxxx
```text
{"timestamp":"2026-08-05T18:49:28.622020Z","level":"info","event":"::endgroup::","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":186}
{"timestamp":"2026-08-05T18:49:29.219279Z","level":"info","event":"[base] tick 0 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:34.224101Z","level":"info","event":"[base] tick 1 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:39.229335Z","level":"info","event":"[base] tick 2 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:44.230741Z","level":"info","event":"[base] tick 3 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:49.235802Z","level":"info","event":"[base] tick 4 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:54.240729Z","level":"info","event":"[base] tick 5 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:59.245828Z","level":"info","event":"[base] tick 6 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:04.247038Z","level":"info","event":"[base] tick 7 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:09.247528Z","level":"info","event":"[base] tick 8 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:14.250719Z","level":"info","event":"[base] tick 9 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:19.255868Z","level":"info","event":"[base] tick 10 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:24.256810Z","level":"info","event":"[base] tick 11 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:29.261832Z","level":"info","event":"[base] tick 12 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:34.266976Z","level":"info","event":"[base] tick 13 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:39.272003Z","level":"info","event":"[base] tick 14 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:44.277099Z","level":"info","event":"[base] tick 15 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:49.282109Z","level":"info","event":"[base] tick 16 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:54.286787Z","level":"info","event":"[base] tick 17 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:59.288915Z","level":"info","event":"[base] tick 18 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:04.293390Z","level":"info","event":"[base] tick 19 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:09.296800Z","level":"info","event":"[base] tick 20 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:14.301846Z","level":"info","event":"[base] tick 21 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:19.306809Z","level":"info","event":"[base] tick 22 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:24.311349Z","level":"info","event":"[base] tick 23 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:29.315332Z","level":"info","event":"[base] tick 24 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:34.316805Z","level":"info","event":"[base] tick 25 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:36.975521Z","level":"info","event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor","filename":"supervisor.py","lineno":1403}
{"timestamp":"2026-08-05T18:51:39.344660Z","level":"info","event":"[base] tick 26 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:44.322040Z","level":"info","event":"[base] tick 27 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
```

Deleting only one of the two pods did **not** reproduce the issue in our testing. In those cases, the task correctly transitioned to `failed` or `up_for_retry`.

**What we observe when it reproduces**

The last log emitted by the per-task pod is:

```text
{"event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor","filename":"supervisor.py","lineno":1403}
```

After that, we never see:

- `on_kill()` being invoked.
- The provider's `"Deleting pod: ..."` log line.
- Any exception or traceback.
- Any further output from the child pod.

Despite the child pod being interrupted before completing its work, the task is later reported by the API server as `success`.

**Observed in production**

We observed this four times across two different DAGs over four days. Every occurrence coincided with our autoscaler reclaiming a Spot/preemptible node hosting one or both pods.

When the issue reproduces:

```text
19:12:28Z [child pod progress log — task actively running]
19:12:42Z
19:12:44Z {"event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor"}

--> Task instance later shows state = success
```

A normal execution of the same task instead includes:

```text
{"event":"Deleting pod: ", ...}
{"event":"::group::Post Execute", ...}
{"event":"Workload finished", "final_state":"success", ...}
```

**Contrast: interrupting only the child pod**

If only the child pod is interrupted while the per-task pod continues running, the task correctly fails. We've observed both `AirflowException` (PodFailed) and `NotFoundException` (404 after pod deletion), but in both cases the task is never reported as `success`.

### What you think should happen instead?

The task should end in `failed` (or `up_for_retry`), just as it does when only the child pod is interrupted. Reporting `success` for a task whose workload never completed is a correctness issue, downstream tasks proceed as if the work succeeded, `on_failure_callback` is never triggered, and no retry occurs.

We haven't identified the root cause and are only reporting the observed, reproducible behavior. This may be related to #58936 (fixed by #61627 in 3.3.0), which changed how `SIGTERM` is forwarded from the supervisor to the task subprocess. The last log we consistently see (`"Received signal, forwarding to task subprocess"`) originates from that change, although we can't confirm whether this is a regression from that fix or a separate issue.

### Operating System

Debian GNU/Linux 12 (bookworm)

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

cncf-kubernetes

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.19.0

### Official Helm Chart version

1.18.0

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the provided KubernetesExecutor and KubernetesPodOperator reproduction, then trace KubernetesPodOperator.execute() through task finalization while both pods are deleted in close succession. The payload names no source file or test; done means the interrupted child pod cannot leave the task instance finalized as SUCCESS and the regression is covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.