missing pod metrics on namespaces with pod-security.kubernetes.io/enforce label
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
Recently we upgraded from 8.14.x to 8.15.3 to fix a bug related to service account tokens. It seems as though there is a change in 8.15.3 where some pod metrics are not being collected from namespaces that have the pod-security.kubernetes.io/enforce label. Specifically we noticed that the `kubernetes.pod.cpu.usage.limit.pct` metrics are missing from these namespaces however there might be others.
Here is an example describe on a namespace that is missing the cpu usage limit pct metrics:
```
Name: namespace_name
Labels: bmap-control-plane=true
kubernetes.io/metadata.name=namespace_name
pod-security.kubernetes.io/enforce=baseline
purpose=namespace_name
Annotations:
Status: Active
No resource quota.
No LimitRange resource.
```
and the only difference I can find between all the other namespaces that are still acting normally is the pod-security label.
If I remove this label metricbeat will start to provide the missing metric. Adding the label back causes the metrics to no longer be shipped.
some examples from grafana of working vs not:


Contributor guide
Research direction
No files, tests, or entry points are named. Start by reproducing the difference in Metricbeat 8.15.3 with the pod-security.kubernetes.io/enforce label present and absent, then trace Kubernetes pod metric collection for the affected namespace. Done means pod metrics including kubernetes.pod.cpu.usage.limit.pct are shipped consistently when the label is present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100