apache / apache/airflow

KubernetesPodOperator with multiple containers hangs if container other than base container is still running

Open
#39,693 6 comments 0 reactions 0 assignees View on GitHub
area:providers good first issue kind:bug provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

main (development)

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

A `KubernetesPodOperator` with the following `full_pod_spec`:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: multi-container-pod
spec:
restartPolicy: Never
containers:
- name: base
image: busybox
command: ["sh", "-c", "echo base will exit after 30 seconds; sleep 30"]
- name: sidecar
image: busybox
command: ["sh", "-c", "echo sidecar running indefinitely; while true; do sleep 3600; done"]
```

will not mark the task as successful after 30 seconds because the `sidecar` will continue to run after the `base` container has succeeded. This happens because the `pod_manager` gets stuck waiting for pod completion. [This if statement returns `False` when istio is not enabled on the pod.](https://github.com/apache/airflow/blob/8b19b78ba50835423a5385a64c31a0e950548147/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L621-L622)

### What you think should happen instead?

The pod should be considered complete when the base container succeeds regardless of whether or not any other containers on the pod are still running.

### How to reproduce

Create a `KubernetesPodOperator` with the `full_pod_spec` provided.

### Operating System

MacOS 14.4.1

### Versions of Apache Airflow Providers

`apache-airflow-providers-cncf-kubernetes==8.0.1`

### Deployment

Other

### Deployment details

Kubernetes on Google Kubernetes Engine. Kubernetes executors and worker pods all run on the same cluster.

### Anything else?

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in airflow/providers/cncf/kubernetes/utils/pod_manager.py at the linked completion check around lines 621-622, and trace how pod completion is determined when Istio is not enabled. Verify the change against the provided multi-container KubernetesPodOperator reproduction: the task should succeed when the base container exits even while the sidecar remains running.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.