elastic / elastic/integrations

[Kubernetes]: API Server Request Counters change depending on which API server pod handled the request

Open
#13,159 8 comments 0 reactions 0 assignees View on GitHub
Integration:kubernetes needs:triage Stalled Team:obs-ds-hosted-services
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

### Integration Name

Kubernetes [kubernetes]

### Dataset Name

kubernetes.apiserver

### Integration Version

1.80.2

### Agent Version

8.17.2

### Agent Output Type

elasticsearch

### Elasticsearch Version

8.17.0

### OS Version and Architecture

Ubuntu 20.04.6 LTS (Focal Fossa) (x86)

### Software/API Version

1.30

### Error Message

_No response_

### Event Original

_No response_

### What did you do?

Elastic Agent is deployed as a DaemonSet in our Kubernetes clusters in AKS, GKE, and EKS.

### What did you see?

The counter field values for the `kubernetes.apiserver` data stream change depending on which API server pod handles the request, and there is no corresponding field/dimension that allows users to identify which API server returned which values.

For example, in AKS the API Server endpoint is a load balanced service running in Azure's own Kubernetes clusters. So, the `service` and `endpoint` return only single addresses, despite it being possible that there are multiple API server pods behind it:
```
$ kubectl get svc,ep kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.253.0.1 443/TCP 65d

NAME ENDPOINTS AGE
endpoints/kubernetes 20.111.33.42:443 65d
```

We can validate that there are multiple API server pods by checking for leases inside of our own AKS cluster:
```
$ kubectl -n kube-system get lease -l apiserver.kubernetes.io/identity=kube-apiserver
NAME HOLDER AGE
apiserver-ix43dchey4wqohc5tofjrs6vym apiserver-ix43dchey4wqohc5tofjrs6vym_0caa3742-b126-4292-b39e-ec5fdbb1962a 2d
apiserver-k63nnxgmldbm2bvj5rtt4hx7mq apiserver-k63nnxgmldbm2bvj5rtt4hx7mq_7ac2fe99-0cab-4d01-b7e0-190076f1af80 2d
apiserver-s6vcckbaevw2fjweana2i6onem apiserver-s6vcckbaevw2fjweana2i6onem_18bc029f-ff23-4b95-b9ec-72790728d028 2d
apiserver-stbk6fiedill356d7lmyaupqge apiserver-stbk6fiedill356d7lmyaupqge_926532d4-6ca5-4c44-a635-d41224205e45 2d
apiserver-ugqlqikauhvmzofm7aren35afu apiserver-ugqlqikauhvmzofm7aren35afu_ff617a76-6fcd-468a-aeee-8ba6cdc61f4e 2d
apiserver-vcxvs4b4s62tfvkq5mf5gpfycu apiserver-vcxvs4b4s62tfvkq5mf5gpfycu_43e34bfb-12f0-40ef-b95b-c5105635d6dc 2d
```

We can further validate that the counter values change (increase + decrease) depending on which API server handled our request. The below shows us retrieving the same time series, note the large range in values depending on which API server handled the request:
```shell
k -n elastic-agent exec elastic-agent-4wq9k -- bash -c 'while true; do curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/metrics -sk | grep apiserver_request_total | grep "code=\"200\""| grep coordination | grep leases | grep PUT ; date -u; sleep 1; done'
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 511969
Wed Mar 12 01:40:41 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 528728
Wed Mar 12 01:40:43 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 515404
Wed Mar 12 01:40:44 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 506675
Wed Mar 12 01:40:46 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 515411
Wed Mar 12 01:40:47 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 506681
Wed Mar 12 01:40:49 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 515421
Wed Mar 12 01:40:50 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 523107
Wed Mar 12 01:40:51 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 523109
Wed Mar 12 01:40:53 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 506696
Wed Mar 12 01:40:54 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 502193
Wed Mar 12 01:40:56 UTC 2025
apiserver_request_total{code="200",component="apiserver",dry_run="",group="coordination.k8s.io",resource="leases",scope="resource",subresource="",verb="PUT",version="v1"} 515445
Wed Mar 12 01:40:57 UTC 2025
```

### What did you expect to see?

API server request counters are monotonically increasing, or there's an additional field that allows us to dimension which API server handled our request.

### Anything else?

This same problem exists on at least GKE and AKS clusters (possibly EKS too), but exactly how the API server endpoints are abstracted from users differs between CSPs.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.