KubernetesJobOperator.get_pods() fails the task instead of waiting for the Job controller to create pods
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
2.11.2
### What happened and how to reproduce it?
`get_pods()` issues `discover_pods_retry_number + 1` (default 4) `list_namespaced_pod` calls back to back with **no delay**, then raises `No pods running with labels ...` if the pods are not visible yet. Raising `discover_pods_retry_number` only adds more immediate GETs — there is no way to wait.
Production trace for one occurrence:
| time (UTC) | event |
|---|---|
| `00:00:23.223` | `job.py` — `Building job job--` |
| `00:00:23.360` | task failed — `AirflowException: No pods running with labels ...` |
| `00:00:23` | Job controller — `SuccessfulCreate: Created pod job---` |
| `00:01:28` | Job controller — `Job completed` |
`kube_job_created` and `kube_pod_created` are both `00:00:23`: the pod existed in the same second the operator gave up, 137 ms after the Job was built. The workload then ran to success unmonitored. ~20 occurrences in 5 days across 8+ unrelated DAGs.
Consequences:
1. The task is marked failed while its Job succeeds. Nothing waits on, monitors or cleans up the Job; `on_kill()` does not fire.
2. `self.pods` is never assigned, so `_cleanup_monitoring_pods()` in the `finally` block no-ops.
3. A retry or manual clear starts a **second** Job while the first is still running or has already committed its writes — silent double execution for non-idempotent tasks.
The message is also inaccurate: the pods are not "not running", they do not exist yet.
### What you think should happen instead?
Wait for the pods, bounded by a timeout. No new parameters needed — `KubernetesJobOperator` extends `KubernetesPodOperator`, so `schedule_timeout_seconds` (deadline, default 120 via `startup_timeout_seconds`) and `startup_check_interval_seconds` (interval, default 5) are already inherited and documented. The first probe stays immediate, so tasks whose pods appear at once are unaffected.
Also: break on `>=` parallelism rather than `==`, and deprecate `discover_pods_retry_number` as a no-op.
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
cncf provider at 10.13.0 but related code is unchanged in current main
### Official Helm Chart version
1.19.0
### Kubernetes Version
1.35.6
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Not Applicable
### Anything else?
The issue is Inherited by `KubernetesStartKueueJobOperator`, `GKEStartJobOperator`, `GKEStartKueueJobOperator` — i.e. exactly the Kueue case where pod creation is slowest.
The issue is known and was raised in https://github.com/apache/airflow/pull/53368 but rejected as out of scope for this PR.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Read job.py's KubernetesJobOperator.get_pods() and the inherited schedule_timeout_seconds and startup_check_interval_seconds behavior first. The fix is done when pod discovery waits within the existing deadline, uses the interval, handles parallelism correctly, and no longer treats delayed creation as missing; account for the inherited operators named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100