apache / apache/airflow

GKEStartJobOperator / KubernetesJobOperator fail to delete pod after it ends

Open
#55,192 3 comments 2 reactions 0 assignees View on GitHub
area:providers kind:bug provider:cncf-kubernetes provider:google Stale Bug Report
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow Provider(s)

cncf-kubernetes

### Versions of Apache Airflow Providers

First seen in
apache-airflow-providers-cncf-kubernetes==10.1.0
Still seeing in
apache-airflow-providers-cncf-kubernetes==10.5.0

I didn't have the issue in 8.3.2

I can't test more recent versions due to versions constraints

### Apache Airflow version

2.10.5

### Operating System

cloud composer

### Deployment

Google Cloud Composer

### Deployment details

_No response_

### What happened

All my task end with
```
[2025-09-02, 15:07:22 EDT] {pod_manager.py:698} INFO - Pod PODNAME has phase Running
[2025-09-02, 15:07:24 EDT] {pod.py:1079} INFO - Skipping deleting pod: PODNAME
```

when the pod correctly succeeds in kubernetes around the same time

```
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 02 Sep 2025 15:05:37 -0400
Finished: Tue, 02 Sep 2025 15:07:21 -0400
```
I always use `OnFinishAction.DELETE_SUCCEEDED_POD`

### What you think should happen instead

Should delete the pod per `OnFinishAction.DELETE_SUCCEEDED_POD`

### How to reproduce

I have this on any task. I use `GKEStartPodOperator` and run python scripts through it. For example:
```
GKEStartPodOperator(
image="bash",
cmds=["/usr/local/bin/bash", "-c"],
arguments=[
"echo",
"hi",
],
)
```

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the pod-cleanup behavior with GKEStartPodOperator or KubernetesJobOperator and the DELETE_SUCCEEDED_POD setting, using the reported provider versions. Trace the operator and pod-manager logs around the transition from Running to Completed. Done means a successfully completed pod is deleted as requested, with coverage for the reported regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, kubernetes, python
Domain
cloud, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.