elastic / elastic/beats

missing pod metrics on namespaces with pod-security.kubernetes.io/enforce label

Open
#42,400 4 comments 0 reactions 0 assignees View on GitHub
needs_team Stalled
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Recently we upgraded from 8.14.x to 8.15.3 to fix a bug related to service account tokens. It seems as though there is a change in 8.15.3 where some pod metrics are not being collected from namespaces that have the pod-security.kubernetes.io/enforce label. Specifically we noticed that the `kubernetes.pod.cpu.usage.limit.pct` metrics are missing from these namespaces however there might be others.

Here is an example describe on a namespace that is missing the cpu usage limit pct metrics:

```
Name: namespace_name
Labels: bmap-control-plane=true
kubernetes.io/metadata.name=namespace_name
pod-security.kubernetes.io/enforce=baseline
purpose=namespace_name
Annotations:
Status: Active

No resource quota.

No LimitRange resource.
```

and the only difference I can find between all the other namespaces that are still acting normally is the pod-security label.

If I remove this label metricbeat will start to provide the missing metric. Adding the label back causes the metrics to no longer be shipped.

some examples from grafana of working vs not:

![Image](https://github.com/user-attachments/assets/fbf1149f-e80d-4231-be26-cc109aa9f439)

![Image](https://github.com/user-attachments/assets/887b0d1e-6ab2-490c-8dcc-3b22c4007960)

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reproducing the difference in Metricbeat 8.15.3 with the pod-security.kubernetes.io/enforce label present and absent, then trace Kubernetes pod metric collection for the affected namespace. Done means pod metrics including kubernetes.pod.cpu.usage.limit.pct are shipped consistently when the label is present.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.