KubernetesPodOperator keeps working by pod name after the durable reattach uid check
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
**Issue Description**
This is a follow up of #71743, and sorry if I miss something here.
In #71744 the pod uid is check one time, when the identity is read back from task state store. That close the reattach case. But after that check the operator keep working by name and namespace only, so if the pod go away and another pod take the same name, the later call can land on the other pod.
Two place I find:
1. `KubernetesPodTrigger` keep only `pod_name` and `pod_namespace`. The constructor and `serialize()` both have no uid, and `_get_pod()` do
```python
pod = await self.hook.get_pod(name=self.pod_name, namespace=self.pod_namespace)
```
on every poll. A deferrable task can wait very long, so this window is not small.
2. `PodManager.delete_pod()` send
```python
self._client.delete_namespaced_pod(
pod.metadata.name, pod.metadata.namespace, body=client.V1DeleteOptions()
)
```
with no preconditions. `KubernetesHook.delete_pod()` and `on_kill()` are the same, they all delete by name.
So the order is like this:
```
operator check the uid, pod A is correct
pod A is gone
pod B take the same name
next poll / log read / delete work on pod B
```
**Steps to reproduce**
I do not race a real trigger poll, sorry. I only show the delete part, on k8s v1.37.0-rc.0 with apache-airflow-providers-cncf-kubernetes 10.21.0. The trigger part I read from the code.
```
the operator observed this pod and remembered it
kpo-uid-delete-test/my-pod uid = 4057eb7b-8483-4064-8779-ecc230861cf1
it goes away and something else takes the name
kpo-uid-delete-test/my-pod uid = 5284f5cd-b90b-4e4c-9f10-a039495a53ea
A. delete the way PodManager.delete_pod does it today
accepted -> the replacement is being deleted
pod gone. the deleted one was uid 5284f5cd-..., not the remembered 4057eb7b-...
B. same delete, but with the remembered uid as a precondition
another replacement appears, uid = c3c80f2a-54e9-4714-a147-d50f90514506
rejected 409 Conflict
still alive: uid c3c80f2a-54e9-4714-a147-d50f90514506
```
script I use
```python
import subprocess, time
from kubernetes import client as k8s, config as k8s_config
NS, POD = "kpo-uid-delete-test", "my-pod"
def body():
return {"apiVersion": "v1", "kind": "Pod",
"metadata": {"name": POD, "namespace": NS},
"spec": {"restartPolicy": "Never", "containers": [
{"name": "base", "image": "registry.k8s.io/pause:3.9"}]}}
def wait_gone(core):
while True:
try:
core.read_namespaced_pod(POD, NS); time.sleep(1)
except k8s.rest.ApiException:
return
subprocess.run(["kubectl", "delete", "ns", NS, "--ignore-not-found"])
subprocess.run(["kubectl", "create", "ns", NS])
k8s_config.load_kube_config()
core = k8s.CoreV1Api()
uid_a = core.create_namespaced_pod(NS, body()).metadata.uid
core.delete_namespaced_pod(POD, NS, body=k8s.V1DeleteOptions(grace_period_seconds=0))
wait_gone(core)
uid_b = core.create_namespaced_pod(NS, body()).metadata.uid
print("remembered", uid_a, "/ now", uid_b)
# A: how delete_pod does it today
core.delete_namespaced_pod(POD, NS, body=k8s.V1DeleteOptions(grace_period_seconds=0))
wait_gone(core)
# B: with the uid as a precondition
uid_c = core.create_namespaced_pod(NS, body()).metadata.uid
try:
core.delete_namespaced_pod(POD, NS, body=k8s.V1DeleteOptions(
grace_period_seconds=0, preconditions=k8s.V1Preconditions(uid=uid_a)))
except k8s.rest.ApiException as e:
print("rejected", e.status, "/ still alive", core.read_namespaced_pod(POD, NS).metadata.uid)
```
### What you think should happen instead?
Once the operator know which pod it work on, I think the uid can travel with it, so every later call talk to the same pod.
For the trigger, `KubernetesPodTrigger` can take a `pod_uid`, put it in `serialize()`, and check it after each `_get_pod()`. If the uid is not the same, the pod we wait for is already gone, and that is a normal end condition, not a new pod to watch.
For the delete, `V1DeleteOptions` already accept `preconditions`, and the client we pin (kubernetes 36.0.3) already have `V1Preconditions` with a `uid` field. A 409 from it just mean the pod we want to delete is not there any more, so we can treat it like the 404 we already handle, and leave the other pod alone. This is nicer than read first and delete after, because the api server do the check for us.
But this change touch the sync path, the async path and the cleanup together, and the trigger serialization is not a small thing to change, so I do not want to guess. May I ask if this direction look ok to you before I try a PR? If you already think about it and decide it is fine as it is now, please tell me and I close this. Thank you very much.
Two more thing I am not sure about, maybe they belong here or maybe not:
- `get_or_create_pod()` persist the uid from the server answer, but it `return pod_request_obj`, so `self.pod` carry no uid. If the uid become the runtime identity then it need to live somewhere.
- #70140 list the pods first and delete by name later. It look like the same shape to me, but it is not my PR so I only mention it.
### Operating System
Ubuntu 26.04 LTS
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
cncf-kubernetes
### Deployment details
kubeadm cluster, k8s v1.37.0-rc.0, single node.
apache-airflow==3.3.1, apache-airflow-providers-cncf-kubernetes==10.21.0, kubernetes==36.0.3
### Anything else?
Thank you for #69914 and for the review on it, the durable execution is very useful for us. This is only about the part after the first check, everything else already work well.
Related: #71743 and #71744.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with KubernetesPodTrigger, including its constructor, serialize(), and _get_pod(), then trace PodManager.delete_pod(), KubernetesHook.delete_pod(), on_kill(), and get_or_create_pod(). Check how the synchronous, asynchronous, and cleanup paths carry pod identity. Done means later polling and deletion remain tied to the original UID, while replacement pods are left untouched.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100