apache / apache/airflow

Support concurrent logs collection for containers in KubernetesPodOperator

Open
#45,061 5 comments 0 reactions 0 assignees View on GitHub
kind:feature needs-triage provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

During the discussion in [PR #43853](https://github.com/apache/airflow/pull/43853) with [@dstandish](https://github.com/dstandish), we noticed that there are two separate implementation flows for collecting logs from init_containers and containers that do similar things.

The current implementation processes logs sequentially, which is correct for init_containers because they are executed one after another—each starting only after the previous one has successfully completed.

However, for regular containers (not init_containers), this logic isn’t fully accurate since they can run in parallel.

Related code are https://github.com/apache/airflow/blob/main/providers/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L617

I suggest to implement concurrent logs collection for containers.

### Use case/motivation

This might cause issues during long-running executions, as we won’t see logs from other containers until one of them has completed.

### Related issues

https://github.com/apache/airflow/pull/43853

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start in providers/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py around line 617, then read PR #43853 for the related implementation discussion. Compare the existing log-collection flows for init_containers and regular containers. Done means regular container logs are collected concurrently while init_container logs remain sequential.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.