apache / apache/airflow

Airflow Not Able to Found the Kubernetive POD looks like API Pool Issue

Open
#47,283 17 comments 0 reactions 0 assignees View on GitHub
_eol_affected_version:2.10 Can't Reproduce kind:bug provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

Other Airflow 2 version (please specify below)

### If "Other Airflow 2 version" selected, which one?

2.10.2

### What happened?

Airflow version: 2.10.2

[2025-03-01, 00:33:13 IST] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 637, in execute_sync
istio_enabled = self.is_istio_enabled(self.pod)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 940, in is_istio_enabled
remote_pod = self.pod_manager.read_pod(pod)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 336, in wrapped_f
return copy(f, *args, **kw)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 475, in call
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 376, in iter
result = action(retry_state)
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 418, in exc_check
raise retry_exc.reraise()
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 185, in reraise
raise self.last_attempt.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/init.py", line 478, in call
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 720, in read_pod
return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23693, in read_namespaced_pod
return self.read_namespaced_pod_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 23780, in read_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '90b2298d-56c6-406b-95fb-661a4bd1942e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'b97d9c8b-7397-47a1-b30e-b99f7c2ae92e', 'X-Kubernetes-Pf-Prioritylevel-Uid': '48b7c182-bee3-4b45-b4dc-a0f80814e25c', 'Date': 'Fri, 28 Feb 2025 19:03:10 GMT', 'Content-Length': '294'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods "co-rl-v2-1-model-update-cb-opt-arm-15-cb-opt-arm-16-v2-r2980b03" not found","reason":"NotFound","details":{"name":"co-rl-v2-1-model-update-cb-opt-arm-15-cb-opt-arm-16-v2-r2980b03","kind":"pods"},"code":404}
During handling of the above exception, another exception occurred:

### What you think should happen instead?

There is the issue in pooling the K8 Pod the pod got deleted before the airflow reach out to it check the status.

### How to reproduce

Create a small K8 pod doing simple things. When the airflow reach out to k8 pod.

### Operating System

linux

### Versions of Apache Airflow Providers

_No response_

### Deployment

Other 3rd-party Helm chart

### Deployment details

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow/providers/cncf/kubernetes/operators/pod.py and airflow/providers/cncf/kubernetes/utils/pod_manager.py, then reproduce the reported case using a small Kubernetes pod. Trace the read_pod call and compare its behavior when the pod has already been deleted; done means the reported failure mode is understood and covered by an appropriate test or confirmed fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.