apache / apache/airflow

KubernetesExecutor: run_pod_async has no retry for transient connection errors (RemoteDisconnected/ProtocolError), crashing the scheduler

Open
#71,879 1 comment 0 reactions 0 assignees View on GitHub
kind:bug needs-triage provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

3.3.1

### What happened and how to reproduce it?

### Apache Airflow version

3.3.1 and 2.11.2 ; reproduced in current `main` of
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py

### What happened

`run_pod_async()` in the KubernetesExecutor calls `create_namespaced_pod()`
with no retry handling for transient network/connection failures. When the
call to the Kubernetes API server fails with a `RemoteDisconnected` /
`ProtocolError` (connection dropped without a response, a common transient
condition on managed clusters, e.g. AKS API server load-balancer resets),
the exception is caught only long enough to emit a stats metric and log,
then unconditionally re-raised:

except ApiException as e:
...
raise
except Exception as e:
...
raise e

This propagates out of `run_pod_async` → `run_next` → `sync()` →
`executor.heartbeat()` → `SchedulerJob._run_scheduler_loop()`, which is not
prepared to treat pod-creation failures as retryable. The scheduler process
then exits (non-zero) and has to be restarted by kubelet.

Observed traceback:

File ".../kubernetes_executor_utils.py", line 452, in run_pod_async
resp = self.kube_client.create_namespaced_pod(
...
urllib3.exceptions.ProtocolError: ('Connection aborted.',
RemoteDisconnected('Remote end closed connection without response'))
...
File ".../scheduler_job_runner.py", in _run_scheduler_loop
executor.heartbeat()
...
[scheduler process exits, kubelet restarts the pod]

I confirmed this same unguarded re-raise pattern is still present on current
`main`, including in the newer async batched pod-creation path
(`_create_pods_async`), which also has no retry, it just isolates the
failure to one job via `asyncio.gather(..., return_exceptions=True)` rather
than crashing the scheduler outright, which is an incidental side effect,
not a fix, and only applies when async batch creation is enabled.

This is a narrower, more specific case of #24748 (configurable retry policy
for the k8s ApiClient), which remains open with no merged fix. This issue is
scoped specifically to the pod-creation call path and its direct impact on
scheduler stability.

### How to reproduce

Inject a connection drop / RST on the path between the scheduler and the
Kubernetes API server while a task pod is being created (e.g. via a chaos
proxy, or simply during periods of API server load-balancer rotation on
AKS). The scheduler will crash on the next `create_namespaced_pod` call that
lands during the drop.

### Deployment details

KubernetesExecutor, standard Astro worker
pod template with git-sync sidecar.

### What you think should happen instead?

A single transient connection drop while creating a task pod should not be
fatal to the scheduler process. `run_pod_async` (and its async counterpart)
should retry `create_namespaced_pod` a small, bounded number of times with
backoff for connection-level errors (e.g. `urllib3.exceptions.ProtocolError`,
`http.client.RemoteDisconnected`, `ConnectionError`), before giving up and
raising.

Either approach should be configurable (max retries, backoff) so operators
running on cloud-managed API servers with occasional connection resets
(AKS, EKS, GKE) don't see routine scheduler restarts from single blips.

### Operating System

Linux (AKS)

### Deployment

Astronomer

### Apache Airflow Provider(s)

cncf-kubernetes

### Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes (10.19.0); same gap confirmed present in
airflow/providers/cncf/kubernetes `main` as of this report.

### Official Helm Chart version

Not Applicable

### Kubernetes Version

NA

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py, focusing on run_pod_async, _create_pods_async, and the create_namespaced_pod call. Trace how failures reach run_next, sync(), and executor.heartbeat(), then inspect existing provider tests and retry-related configuration. Done means transient connection failures receive bounded, configurable retries with backoff without making the scheduler exit.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.