KubernetesPodOperator: init_container_logs is silently ignored when deferrable=True
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
`KubernetesPodOperator.init_container_logs` only has an effect when `deferrable=False`.
In the sync execution path, `execute_sync` calls `await_pod_start` and then `await_init_containers_completion` (operators/pod.py), which — when `init_container_logs` is set — calls
`pod_manager.fetch_requested_init_container_logs(...)` to stream the init containers' logs.
In the deferrable path, `execute_async`/`invoke_defer_method` constructs a `KubernetesPodTrigger` and passes it `get_logs`, `base_container_name`, `poll_interval`, etc., but never passes `init_container_logs` or `init_containers` — the trigger has no such parameter at all. `triggers/pod.py` has no references to init containers anywhere; it only fetches logs for `base_container_name` (the main container) via `fetch_container_logs_before_current_sec`. `trigger_reentry`, which runs after the task resumes from deferral, also never calls `await_init_containers_completion`.
As a result, setting `init_container_logs` on the operator has no effect once `deferrable=True` is set — the parameter is silently accepted but ignored, and no init container logs appear in the task logs at all.
### What you think should happen instead?
Either:
- When `init_container_logs` is set to True, `KubernetesPodTrigger` should be extended to fetch/stream logs for all the init containers (mirroring the sync-path behavior in `await_init_containers_completion` / `fetch_requested_init_container_logs`), or
- We add a new flag `init_container_logs_when_deferrable`, and let `KubernetesPodTrigger` allow fetch/stream logs for all the init containers
### Operating System
N/A
### Deployment
None
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
latest
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
Related but not duplicate:
- #42498 fixed init container log support for the **sync** path only (fix landed in #43853); it does not mention or address the deferrable path.
- #56895 is a broader, unrelated feature request about efficiency of log streaming in deferred mode in general.
Relevant code:
- `providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py`
(`execute_sync`, `await_init_containers_completion`, `invoke_defer_method`)
- `providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py`
(`KubernetesPodTrigger.__init__`, `serialize`, `run`)
- `providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py`
(`fetch_requested_init_container_logs`, `get_init_container_names`)
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in operators/pod.py by comparing execute_sync, invoke_defer_method, and trigger_reentry, then inspect KubernetesPodTrigger in triggers/pod.py and the init-container helpers in utils/pod_manager.py. Extend the deferrable path according to the selected behavior so init_container_logs is honored and logs from all init containers appear; verify the existing sync behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100