KubernetesExecutor: delete_pod fires before the self.running dedup check, causing repeated deletes per TaskInstanceKey
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
KubernetesExecutor issues 2-3 delete_pod API calls for the same finished worker pod instead of one. The first call succeeds; the rest return 404 from the Kubernetes API. All calls come from the same scheduler process for the same TaskInstanceKey, milliseconds apart.
Observed on a production-scale cluster: over a 15-minute sample, 306 pods received 3 delete calls, 15 received 2, and only 4 received the intended 1. That's ~1.47 delete calls per pod and a 67% 404 rate, generating a large amount of unnecessary kube-apiserver audit-log volume (and associated CloudWatch cost on EKS) at scale. Mapped tasks (non-null map_index) appear to be the common factor in every case we captured.
Root cause
In the result-processing path (kubernetes_executor.py / kubernetes_executor_utils.py in apache-airflow-providers-cncf-kubernetes), delete_pod is invoked before the executor's self.running dedup check runs for that TaskInstanceKey. Because the same task-completion event gets processed more than once before the dedup check has a chance to short-circuit it, the pod delete is issued redundantly on each pass.
### What you think should happen instead?
The dedup check against self.running should happen (or the running-set be updated) before delete_pod is called, so a given TaskInstanceKey only ever triggers one delete call, regardless of how many times its completion event is processed.
### Operating System
_No response_
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==10.21.0
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in the result-processing path in kubernetes_executor.py and kubernetes_executor_utils.py, tracing where delete_pod is called relative to the self.running dedup check for mapped TaskInstanceKey values. Confirm the change prevents repeated completion processing from issuing more than one delete_pod call, including avoiding the observed repeated 404 responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100