KubernetesExecutor: surface pod log tail for generic Failed-state pods
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Apache Airflow version
main (3.x)
### Description
When a KubernetesExecutor-managed worker pod terminates with `exit code 1, reason: "Error"` and no `message` set on the container status (the common case for Python import errors at task-runner startup), `AirflowKubernetesScheduler._get_pod_failure_reason()` returns a string like:
```
Pod base reason: Error
```
This is the entire payload that makes it into the scheduler log and the task event_buffer info field. Operators then have to chase the pod logs out-of-band (kubectl, audit logs, log aggregation pipeline) to find the actual traceback.
### Use case / motivation
For "generic" failures (no `container.status.message`), optionally append the last N lines of the pod's logs to the failure reason string. Two new opt-in config keys on `[kubernetes_executor]`:
- `failure_pod_log_lines` (int, default 0 = disabled, recommended 50-100)
- `failure_log_read_timeout` (int seconds, default 5)
When `failure_pod_log_lines > 0` and the failure is "generic", call `CoreV1Api.read_namespaced_pod_log(..., tail_lines=N, _request_timeout=T)` and append the result. Wrap in try/except so a read-log failure never propagates out of the failure handler.
### Operational context
What we run today at LinkedIn DI is a local enhancement of `_get_pod_failure_reason` that walks both init-container and main-container statuses, formats per-container `exit code / reason / message`, and falls back to `pod.status.reason` / `pod.status.message` when no container yielded a clear reason. That covers the structured failures (container-status `terminated` or `waiting` with a populated reason string) well.
What it doesn't cover — and what this proposal targets — is the generic `exit_code=1, reason="Error"` case with no container `message`. Python import errors at task-runner startup, OOM-killers landing on PID 1, malformed entrypoint shells. The container terminates cleanly enough that the K8s API has nothing structured to report, but the pod logs contain the actual traceback. Today operators chase those out-of-band (kubectl logs, log aggregator); piping the last N lines of pod log into the failure reason when the structured path is empty is what closes that residual gap.
### Related issues
I have not found a tracking issue for this; happy to be pointed at one if it exists.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Contributor guide
Assessment
This issue has not been assessed yet.