Need to log hostname as metadata, not just for celery
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Body
### user story
as a support engineer, i want to be able to see which pod a task instance ran on for every task attempt
### more detail
Airflow tracks the hostname field on the TI table. It's purpose there is mainly for reading logs of running tasks served over http server on the celery worker.
Because this is its purpose, we allow users to override the hostname callable to be something else, e.g. the IP address of the worker.
But this tends to conflict with its other use, namely of troubleshooting and debugging. In the k8s context, the pod name can be more useful than the IP address. So if we're using IP address instead of pod name, then we lose some troubleshooting ability.
### suggestion
let's maybe add this info in the event log. maybe the Log table should add an extra json field with arbitrary metadata info
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Research direction
Start by tracing how the hostname is stored on the task-instance table and how event and task logs are recorded. Compare the hostname's log-serving purpose with its troubleshooting use, then define where pod identity and other metadata should be stored and how support engineers would retrieve it. No files or tests are named, so the affected entry points and completion checks need to be established first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100