elastic / elastic/integrations
[kubernetes]: leader election issue with Kube-state-metrics
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Integration Name
Kubernetes [kubernetes]
### Dataset Name
All kube state metrics related
### Integration Version
1.58.0
### Agent Version
8.15.3
### Agent Output Type
elasticsearch
### Elasticsearch Version
8.15.3
### OS Version and Architecture
Ubuntu
### Software/API Version
_No response_
### Error Message
No new kube state metrics piped to elastic out of the blue. No changes from infra, kubernetes cluster, elastic integration or any sort. when we disable leader election, it started piping but all daemonset set elastic agent are piping which we do not want that, and after toggling on leader election again, no new metrics are coming in again.
Below was the diagnosis i tried to dig
This was the node health i.e. the leader elected node. Seems healthy from what i see.
```
~/gi/p/tmp kc get lease
NAME HOLDER AGE
elastic-agent-cluster-leader elastic-agent-leader-elastic-agent-vkkqs 480d
~/gi/p/tmp kc describe lease/elastic-agent-cluster-leader
Name: elastic-agent-cluster-leader
Namespace: elastic-agent
Labels:
Annotations:
API Version: coordination.k8s.io/v1
Kind: Lease
Metadata:
Creation Timestamp: 2024-05-30T03:26:35Z
Resource Version: 646691471
UID: 77dfa113-82b7-46b3-8043-112aa4ee7fc9
Spec:
Acquire Time: 2025-09-16T13:30:47.957845Z
Holder Identity: elastic-agent-leader-elastic-agent-vkkqs
Lease Duration Seconds: 15
Lease Transitions: 39
Renew Time: 2025-09-23T03:12:14.654571Z
Events:
~/gi/p/tmp kc delete lease/elastic-agent-cluster-leader
```
checking the last few moment of agent error logs, i spotted it attempted to hit kube state metric endpoint, maybe it was unavailable for a moment, and maybe it just gave up thereafter.
### Event Original
_No response_
### What did you do?
deleting the lease solved the problem as it recreate a new lease and metrics started piping.
I had similar cluster also hitting the same problem last month, we tried restarting all elastic agent but it did not help and it recovered a few days later.
thought maybe it was integration version issue, which i have then upgraded to 1.81.X (latest) but it did not recover
### What did you see?
No new kube state metrics piped
### What did you expect to see?
Kube state metrics to pipe consistently. Even when kube state metrics is unavailable, it should resume once the connectivity is open up
### Anything else?
Kubernetes EKS 1.32
No istio/service mesh enabled for the namespace
Contributor guide
Assessment
This issue has not been assessed yet.