Support concurrent logs collection for containers in KubernetesPodOperator
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
During the discussion in [PR #43853](https://github.com/apache/airflow/pull/43853) with [@dstandish](https://github.com/dstandish), we noticed that there are two separate implementation flows for collecting logs from init_containers and containers that do similar things.
The current implementation processes logs sequentially, which is correct for init_containers because they are executed one after another—each starting only after the previous one has successfully completed.
However, for regular containers (not init_containers), this logic isn’t fully accurate since they can run in parallel.
Related code are https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L617
I suggest to implement concurrent logs collection for containers.
### Use case/motivation
This might cause issues during long-running executions, as we won’t see logs from other containers until one of them has completed.
### Related issues
https://github.com/apache/airflow/pull/43853
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start in providers/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py around line 617, then read PR #43853 for the related implementation discussion. Compare the existing log-collection flows for init_containers and regular containers. Done means regular container logs are collected concurrently while init_container logs remain sequential.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100