apache / apache/airflow

KubernetesExecutor: delete_pod fires before the self.running dedup check, causing repeated deletes per TaskInstanceKey

Open
#72,803 1 comment 1 reaction 0 assignees View on GitHub
area:providers kind:bug provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

3.3.1

### What happened and how to reproduce it?

KubernetesExecutor issues 2-3 delete_pod API calls for the same finished worker pod instead of one. The first call succeeds; the rest return 404 from the Kubernetes API. All calls come from the same scheduler process for the same TaskInstanceKey, milliseconds apart.

Observed on a production-scale cluster: over a 15-minute sample, 306 pods received 3 delete calls, 15 received 2, and only 4 received the intended 1. That's ~1.47 delete calls per pod and a 67% 404 rate, generating a large amount of unnecessary kube-apiserver audit-log volume (and associated CloudWatch cost on EKS) at scale. Mapped tasks (non-null map_index) appear to be the common factor in every case we captured.

Root cause

In the result-processing path (kubernetes_executor.py / kubernetes_executor_utils.py in apache-airflow-providers-cncf-kubernetes), delete_pod is invoked before the executor's self.running dedup check runs for that TaskInstanceKey. Because the same task-completion event gets processed more than once before the dedup check has a chance to short-circuit it, the pod delete is issued redundantly on each pass.

### What you think should happen instead?

The dedup check against self.running should happen (or the running-set be updated) before delete_pod is called, so a given TaskInstanceKey only ever triggers one delete call, regardless of how many times its completion event is processed.

### Operating System

_No response_

### Deployment

None

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.21.0

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in the result-processing path in kubernetes_executor.py and kubernetes_executor_utils.py, tracing where delete_pod is called relative to the self.running dedup check for mapped TaskInstanceKey values. Confirm the change prevents repeated completion processing from issuing more than one delete_pod call, including avoiding the observed repeated 404 responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.