elastic / elastic/apm

Enhance our k8s pod detection

Open
#881 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

Today agents support sending k8s data if the user explicitly configures pods to be injected with` KUBERNETES_NAMESPACE, KUBERNETES_POD_NAME, KUBERNETES_POD_UID, KUBERNETES_NODE_NAME`: https://www.elastic.co/guide/en/observability/current/apm-api-metadata.html#apm-api-kubernetes-data and https://github.com/elastic/apm/issues/21#issue-383877963

Whilst discussing if we can [improve our `host.name` detection here](https://github.com/elastic/observability-dev/issues/3321#issuecomment-2236690064), I noticed the following environment variable should be exposed to all pods natively

`KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT`

@trentm also pointed out `kubectl` itself relies on these to observe its running inside a pod: https://kubernetes.io/docs/reference/kubectl/#in-cluster-authentication-and-namespace-overrides

The suggestion here to start reporting these as agents and to [extend the apm intake protocol](
https://github.com/elastic/apm-data/blob/main/model/modeljson/internal/kubernetes.go#L20-L24) to actually ingest this information into Elasticsearch.

Lastly we currently set `host.name` to `kubernetes.host_name` we detect from `KUBERNETES_NODE_NAME`: https://github.com/elastic/apm-data/blob/main/model/modelprocessor/hostname.go#L39

However we don't record the inverse, we are running under k8s but `KUBERNETES_NODE_NAME` was not explicitly configured. I believe we need to record this flag in Elasticsearch so that the Hosts View can filter this data out correctly to fix: https://github.com/elastic/observability-dev/issues/3321.

Sending and recording `KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT` would allow us to detect and record that flag.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with apm-data/model/modeljson/internal/kubernetes.go and apm-data/model/modelprocessor/hostname.go, then trace how Kubernetes metadata is sent through the APM intake protocol. Confirm which agents expose KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT and how the running-under-Kubernetes flag should be represented in Elasticsearch. Done means the new metadata is ingested and the Hosts View can distinguish Kubernetes-hosted data.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, kubernetes
Domain
backend-api-design, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.