Mirantis / Mirantis/cri-dockerd
Missing old metrics in cadvisor
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 356
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 3
Description
After doing a kubernetes upgrade from v1.23 to v1.24, some metric labels have gone missing from a grafana dashboard. We are using cri-dockerd for docker CRI.
Previously (kubernetes v1.23.3)
container_memory_swap{container="kube-scheduler",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3da42828c928b12552def61111bf21f.slice/docker-194b27fff8dbca592c0d8cbb22bb79bd8fe159ce5329187f4c27b774d46a244b.scope",image="sha256:a4183b88f6e65972c4b176b43ca59de31868635a7e43805f4c6e26203de1742f",name="k8s_kube-scheduler_kube-scheduler-master-worker-07b03d19518164ffb42f_kube-system_f3da42828c928b12552def61111bf21f_0",namespace="kube-system",pod="kube-scheduler-master-worker-07b03d19518164ffb42f"} 0 1669121874089
Currently (kubernetes v1.24.6)
container_memory_swap{container="",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52d0cf9b11e9c1a9b69391aa1c91ac3.slice",image="",name="",namespace="kube-system",pod="kube-scheduler-master-worker-91ea15c226fe61b78e7e"} 0 1669122010125
Notice how the "container" label for this metric changed from a value to an empty string.
We have currently upgraded to the following components/versions:
Kubernetes: v1.24.6
Docker: 20.10.18
cri-dockerd: 0.2.6 (d8accf7)
cadvisor: v0.44.1
The commands we used to get the metrics were:
NODE_NAME=$(kubectl get no --no-headers | cut -f 1 -d ' ')
kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/metrics/cadvisor | grep container_memory_swap | grep kube-scheduler
We found a very similar issue on the rancher boards:
https://github.com/rancher/rancher/issues/38934
If I can do anything to help with information further, please let me know.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the report with Kubernetes v1.24.6, Docker 20.10.18, cri-dockerd 0.2.6, and cadvisor v0.44.1 using the kubectl metrics command shown in the issue. Compare the resulting labels with the v1.23 example and investigate whether cri-dockerd is responsible; done means identifying the cause and restoring the expected metric labels or documenting the affected compatibility boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- devops, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100