Airflow Not Able to Found the Kubernetive POD looks like API Pool Issue
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.2
### What happened?
Airflow version: 2.10.2
[2025-03-01, 00:33:13 IST] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 637, in execute_sync
istio_enabled = self.is_istio_enabled(self.pod)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 940, in is_istio_enabled
remote_pod = self.pod_manager.read_pod(pod)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 336, in wrapped_f
return copy(f, *args, **kw)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 475, in call
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 376, in iter
result = action(retry_state)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 418, in exc_check
raise retry_exc.reraise()
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 185, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 478, in call
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 720, in read_pod
return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23693, in read_namespaced_pod
return self.read_namespaced_pod_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23780, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '90b2298d-56c6-406b-95fb-661a4bd1942e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'b97d9c8b-7397-47a1-b30e-b99f7c2ae92e', 'X-Kubernetes-Pf-Prioritylevel-Uid': '48b7c182-bee3-4b45-b4dc-a0f80814e25c', 'Date': 'Fri, 28 Feb 2025 19:03:10 GMT', 'Content-Length': '294'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods "co-rl-v2-1-model-update-cb-opt-arm-15-cb-opt-arm-16-v2-r2980b03" not found","reason":"NotFound","details":{"name":"co-rl-v2-1-model-update-cb-opt-arm-15-cb-opt-arm-16-v2-r2980b03","kind":"pods"},"code":404}
During handling of the above exception, another exception occurred:
### What you think should happen instead?
There is the issue in pooling the K8 Pod the pod got deleted before the airflow reach out to it check the status.
### How to reproduce
Create a small K8 pod doing simple things. When the airflow reach out to k8 pod.
### Operating System
linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with airflow/providers/cncf/kubernetes/operators/pod.py and airflow/providers/cncf/kubernetes/utils/pod_manager.py, then reproduce the reported case using a small Kubernetes pod. Trace the read_pod call and compare its behavior when the pod has already been deleted; done means the reported failure mode is understood and covered by an appropriate test or confirmed fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100