KubernetesPodOperator with multiple containers hangs if container other than base container is still running
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
A `KubernetesPodOperator` with the following `full_pod_spec`:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: multi-container-pod
spec:
restartPolicy: Never
containers:
- name: base
image: busybox
command: ["sh", "-c", "echo base will exit after 30 seconds; sleep 30"]
- name: sidecar
image: busybox
command: ["sh", "-c", "echo sidecar running indefinitely; while true; do sleep 3600; done"]
```
will not mark the task as successful after 30 seconds because the `sidecar` will continue to run after the `base` container has succeeded. This happens because the `pod_manager` gets stuck waiting for pod completion. [This if statement returns `False` when istio is not enabled on the pod.](https://github.com/apache/airflow/blob/8b19b78ba50835423a5385a64c31a0e950548147/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L621-L622)
### What you think should happen instead?
The pod should be considered complete when the base container succeeds regardless of whether or not any other containers on the pod are still running.
### How to reproduce
Create a `KubernetesPodOperator` with the `full_pod_spec` provided.
### Operating System
MacOS 14.4.1
### Versions of Apache Airflow Providers
`apache-airflow-providers-cncf-kubernetes==8.0.1`
### Deployment
Other
### Deployment details
Kubernetes on Google Kubernetes Engine. Kubernetes executors and worker pods all run on the same cluster.
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in airflow/providers/cncf/kubernetes/utils/pod_manager.py at the linked completion check around lines 621-622, and trace how pod completion is determined when Istio is not enabled. Verify the change against the provided multi-container KubernetesPodOperator reproduction: the task should succeed when the base container exits even while the sidecar remains running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100