KubernetesExecutor: run_pod_async has no retry for transient connection errors (RemoteDisconnected/ProtocolError), crashing the scheduler
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
### Apache Airflow version
3.3.1 and 2.11.2 ; reproduced in current `main` of
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
### What happened
`run_pod_async()` in the KubernetesExecutor calls `create_namespaced_pod()`
with no retry handling for transient network/connection failures. When the
call to the Kubernetes API server fails with a `RemoteDisconnected` /
`ProtocolError` (connection dropped without a response, a common transient
condition on managed clusters, e.g. AKS API server load-balancer resets),
the exception is caught only long enough to emit a stats metric and log,
then unconditionally re-raised:
except ApiException as e:
...
raise
except Exception as e:
...
raise e
This propagates out of `run_pod_async` → `run_next` → `sync()` →
`executor.heartbeat()` → `SchedulerJob._run_scheduler_loop()`, which is not
prepared to treat pod-creation failures as retryable. The scheduler process
then exits (non-zero) and has to be restarted by kubelet.
Observed traceback:
File ".../kubernetes_executor_utils.py", line 452, in run_pod_async
resp = self.kube_client.create_namespaced_pod(
...
urllib3.exceptions.ProtocolError: ('Connection aborted.',
RemoteDisconnected('Remote end closed connection without response'))
...
File ".../scheduler_job_runner.py", in _run_scheduler_loop
executor.heartbeat()
...
[scheduler process exits, kubelet restarts the pod]
I confirmed this same unguarded re-raise pattern is still present on current
`main`, including in the newer async batched pod-creation path
(`_create_pods_async`), which also has no retry, it just isolates the
failure to one job via `asyncio.gather(..., return_exceptions=True)` rather
than crashing the scheduler outright, which is an incidental side effect,
not a fix, and only applies when async batch creation is enabled.
This is a narrower, more specific case of #24748 (configurable retry policy
for the k8s ApiClient), which remains open with no merged fix. This issue is
scoped specifically to the pod-creation call path and its direct impact on
scheduler stability.
### How to reproduce
Inject a connection drop / RST on the path between the scheduler and the
Kubernetes API server while a task pod is being created (e.g. via a chaos
proxy, or simply during periods of API server load-balancer rotation on
AKS). The scheduler will crash on the next `create_namespaced_pod` call that
lands during the drop.
### Deployment details
KubernetesExecutor, standard Astro worker
pod template with git-sync sidecar.
### What you think should happen instead?
A single transient connection drop while creating a task pod should not be
fatal to the scheduler process. `run_pod_async` (and its async counterpart)
should retry `create_namespaced_pod` a small, bounded number of times with
backoff for connection-level errors (e.g. `urllib3.exceptions.ProtocolError`,
`http.client.RemoteDisconnected`, `ConnectionError`), before giving up and
raising.
Either approach should be configurable (max retries, backoff) so operators
running on cloud-managed API servers with occasional connection resets
(AKS, EKS, GKE) don't see routine scheduler restarts from single blips.
### Operating System
Linux (AKS)
### Deployment
Astronomer
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes (10.19.0); same gap confirmed present in
airflow/providers/cncf/kubernetes `main` as of this report.
### Official Helm Chart version
Not Applicable
### Kubernetes Version
NA
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py, focusing on run_pod_async, _create_pods_async, and the create_namespaced_pod call. Trace how failures reach run_next, sync(), and executor.heartbeat(), then inspect existing provider tests and retry-related configuration. Done means transient connection failures receive bounded, configurable retries with backoff without making the scheduler exit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100