Kubernetes Launchers Log Check in Time is Not Configurable - Excess Logs Slow Airflow
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
The Kube pod monitor runs every second (+overhead) as referenced below:
https://github.com/apache/airflow/blob/31b956c6c22476d109c45c99d8a325c5c1e0fd45/airflow/kubernetes/pod_launcher.py#L137
Whilst this is ok in many situations, any slow, long running job ends up with excessive logs, it would be useful to be able to pass through an argument to set the log refresh frequency.
Contributor guide
Research direction
Start with airflow/kubernetes/pod_launcher.py at the linked log-check section around line 137, then inspect nearby tests or launcher call sites to understand how the monitor is invoked. Done means a caller can configure the log refresh frequency and long-running jobs no longer produce unnecessary check-in logs at the default interval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100