Enhance our k8s pod detection
- Dominant language
- Gherkin
- Stars
- 427
- Forks
- 125
- PR merge metrics
- No merged PRs in 30d
Description
Today agents support sending k8s data if the user explicitly configures pods to be injected with` KUBERNETES_NAMESPACE, KUBERNETES_POD_NAME, KUBERNETES_POD_UID, KUBERNETES_NODE_NAME`: https://www.elastic.co/guide/en/observability/current/apm-api-metadata.html#apm-api-kubernetes-data and https://github.com/elastic/apm/issues/21#issue-383877963
Whilst discussing if we can [improve our `host.name` detection here](https://github.com/elastic/observability-dev/issues/3321#issuecomment-2236690064), I noticed the following environment variable should be exposed to all pods natively
`KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT`
@trentm also pointed out `kubectl` itself relies on these to observe its running inside a pod: https://kubernetes.io/docs/reference/kubectl/#in-cluster-authentication-and-namespace-overrides
The suggestion here to start reporting these as agents and to [extend the apm intake protocol](
https://github.com/elastic/apm-data/blob/main/model/modeljson/internal/kubernetes.go#L20-L24) to actually ingest this information into Elasticsearch.
Lastly we currently set `host.name` to `kubernetes.host_name` we detect from `KUBERNETES_NODE_NAME`: https://github.com/elastic/apm-data/blob/main/model/modelprocessor/hostname.go#L39
However we don't record the inverse, we are running under k8s but `KUBERNETES_NODE_NAME` was not explicitly configured. I believe we need to record this flag in Elasticsearch so that the Hosts View can filter this data out correctly to fix: https://github.com/elastic/observability-dev/issues/3321.
Sending and recording `KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT` would allow us to detect and record that flag.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with apm-data/model/modeljson/internal/kubernetes.go and apm-data/model/modelprocessor/hostname.go, then trace how Kubernetes metadata is sent through the APM intake protocol. Confirm which agents expose KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT and how the running-under-Kubernetes flag should be represented in Elasticsearch. Done means the new metadata is ingested and the Hosts View can distinguish Kubernetes-hosted data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, kubernetes
- Domain
- backend-api-design, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100