apache / apache/airflow

KubernetesJobOperator.get_pods() fails the task instead of waiting for the Job controller to create pods

Open
#70,585 2 comments 0 reactions 0 assignees View on GitHub
area:providers kind:bug needs-triage provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Providers

### Apache Airflow version

2.11.2

### What happened and how to reproduce it?

`get_pods()` issues `discover_pods_retry_number + 1` (default 4) `list_namespaced_pod` calls back to back with **no delay**, then raises `No pods running with labels ...` if the pods are not visible yet. Raising `discover_pods_retry_number` only adds more immediate GETs — there is no way to wait.

Production trace for one occurrence:

| time (UTC) | event |
|---|---|
| `00:00:23.223` | `job.py` — `Building job job--` |
| `00:00:23.360` | task failed — `AirflowException: No pods running with labels ...` |
| `00:00:23` | Job controller — `SuccessfulCreate: Created pod job---` |
| `00:01:28` | Job controller — `Job completed` |

`kube_job_created` and `kube_pod_created` are both `00:00:23`: the pod existed in the same second the operator gave up, 137 ms after the Job was built. The workload then ran to success unmonitored. ~20 occurrences in 5 days across 8+ unrelated DAGs.

Consequences:

1. The task is marked failed while its Job succeeds. Nothing waits on, monitors or cleans up the Job; `on_kill()` does not fire.
2. `self.pods` is never assigned, so `_cleanup_monitoring_pods()` in the `finally` block no-ops.
3. A retry or manual clear starts a **second** Job while the first is still running or has already committed its writes — silent double execution for non-idempotent tasks.

The message is also inaccurate: the pods are not "not running", they do not exist yet.

### What you think should happen instead?

Wait for the pods, bounded by a timeout. No new parameters needed — `KubernetesJobOperator` extends `KubernetesPodOperator`, so `schedule_timeout_seconds` (deadline, default 120 via `startup_timeout_seconds`) and `startup_check_interval_seconds` (interval, default 5) are already inherited and documented. The first probe stays immediate, so tasks whose pods appear at once are unaffected.

Also: break on `>=` parallelism rather than `==`, and deprecate `discover_pods_retry_number` as a no-op.

### Operating System

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

cncf provider at 10.13.0 but related code is unchanged in current main

### Official Helm Chart version

1.19.0

### Kubernetes Version

1.35.6

### Helm Chart configuration

Not Applicable

### Docker Image customizations

Not Applicable

### Anything else?

The issue is Inherited by `KubernetesStartKueueJobOperator`, `GKEStartJobOperator`, `GKEStartKueueJobOperator` — i.e. exactly the Kueue case where pod creation is slowest.

The issue is known and was raised in https://github.com/apache/airflow/pull/53368 but rejected as out of scope for this PR.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Read job.py's KubernetesJobOperator.get_pods() and the inherited schedule_timeout_seconds and startup_check_interval_seconds behavior first. The fix is done when pod discovery waits within the existing deadline, uses the interval, handles parallelism correctly, and no longer treats delayed creation as missing; account for the inherited operators named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.