apache / apache/airflow

KubernetesExecutor: surface pod log tail for generic Failed-state pods

Open
#66,795 0 comments 0 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

main (3.x)

### Description

When a KubernetesExecutor-managed worker pod terminates with `exit code 1, reason: "Error"` and no `message` set on the container status (the common case for Python import errors at task-runner startup), `AirflowKubernetesScheduler._get_pod_failure_reason()` returns a string like:

```
Pod base reason: Error
```

This is the entire payload that makes it into the scheduler log and the task event_buffer info field. Operators then have to chase the pod logs out-of-band (kubectl, audit logs, log aggregation pipeline) to find the actual traceback.

### Use case / motivation

For "generic" failures (no `container.status.message`), optionally append the last N lines of the pod's logs to the failure reason string. Two new opt-in config keys on `[kubernetes_executor]`:

- `failure_pod_log_lines` (int, default 0 = disabled, recommended 50-100)
- `failure_log_read_timeout` (int seconds, default 5)

When `failure_pod_log_lines > 0` and the failure is "generic", call `CoreV1Api.read_namespaced_pod_log(..., tail_lines=N, _request_timeout=T)` and append the result. Wrap in try/except so a read-log failure never propagates out of the failure handler.

### Operational context

What we run today at LinkedIn DI is a local enhancement of `_get_pod_failure_reason` that walks both init-container and main-container statuses, formats per-container `exit code / reason / message`, and falls back to `pod.status.reason` / `pod.status.message` when no container yielded a clear reason. That covers the structured failures (container-status `terminated` or `waiting` with a populated reason string) well.

What it doesn't cover — and what this proposal targets — is the generic `exit_code=1, reason="Error"` case with no container `message`. Python import errors at task-runner startup, OOM-killers landing on PID 1, malformed entrypoint shells. The container terminates cleanly enough that the K8s API has nothing structured to report, but the pod logs contain the actual traceback. Today operators chase those out-of-band (kubectl logs, log aggregator); piping the last N lines of pod log into the failure reason when the structured path is empty is what closes that residual gap.

### Related issues

I have not found a tracking issue for this; happy to be pointed at one if it exists.

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.