apache / apache/airflow

Need to log hostname as metadata, not just for celery

Open
#54,143 5 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature kind:meta
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Body

### user story

as a support engineer, i want to be able to see which pod a task instance ran on for every task attempt

### more detail

Airflow tracks the hostname field on the TI table. It's purpose there is mainly for reading logs of running tasks served over http server on the celery worker.

Because this is its purpose, we allow users to override the hostname callable to be something else, e.g. the IP address of the worker.

But this tends to conflict with its other use, namely of troubleshooting and debugging. In the k8s context, the pod name can be more useful than the IP address. So if we're using IP address instead of pod name, then we lose some troubleshooting ability.

### suggestion

let's maybe add this info in the event log. maybe the Log table should add an extra json field with arbitrary metadata info

### Committer

- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the hostname is stored on the task-instance table and how event and task logs are recorded. Compare the hostname's log-serving purpose with its troubleshooting use, then define where pod identity and other metadata should be stored and how support engineers would retrieve it. No files or tests are named, so the affected entry points and completion checks need to be established first.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.