nestybox / nestybox/sysbox

Kubelet "cri_stats_provider" logs generated after CRI-O installation

Open
#292 0 comments 1 reaction 1 assignee View on GitHub

@rodnymolina is already working on this.

Since May 22, 2021.

bug documentation
Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

Let me start by saying that his is not a Sysbox issue, but I'm reporting it here because I'm expecting users to reproduce this behavior once they switch to CRI-O runtime as part of the Sysbox installation process in K8s clusters.

As part of the installation of the crio-deploy-k8s daemonset in the elected k8s nodes, kubelet will be restarted and the pre-existing pods will be re-instantiated. From that moment on, kubelet may start to dump logs like these ones every 10 seconds:

May 22 17:10:36 ip-172-20-42-62 kubelet[15230]: E0522 17:10:36.179214   15230 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containers/storage/overlay/d55f04c7ee1011670bae419c7d9d7a3a9d9fb620e01f7e61399a7950052ace5b/merged": failed to get device for dir "/var/lib/containers/storage/overlay/d55f04c7ee1011670bae419c7d9d7a3a9d9fb620e01f7e61399a7950052ace5b/merged": could not find device with major: 0, minor: 93 in cached partitions map.
May 22 17:10:36 ip-172-20-42-62 kubelet[15230]: E0522 17:10:36.188714   15230 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containers/storage/overlay/58197d684779f82964137f1bc50201322cfd8567cad0fd00e51f4dcaf11db28d/merged": failed to get device for dir "/var/lib/containers/storage/overlay/58197d684779f82964137f1bc50201322cfd8567cad0fd00e51f4dcaf11db28d/merged": could not find device with major: 0, minor: 88 in cached partitions map.
May 22 17:10:36 ip-172-20-42-62 kubelet[15230]: E0522 17:10:36.191654   15230 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/containers/storage/overlay/3b6a438c4e5f0bbec943efd47f08773dcee925c0f32e08fcbfb2efa919f22462/merged": failed to get device for dir "/var/lib/containers/storage/overlay/3b6a438c4e5f0bbec943efd47f08773dcee925c0f32e08fcbfb2efa919f22462/merged": could not find device with major: 0, minor: 79 in cached partitions map.

Basically, kubelet's cadvisor/cri-stats component is complaining about being unable to reach overlayfs partitions associated to the pods running in this k8s node. The kubelet task being impacted (cri_stats_provider) is simply collecting system statistics, so this issue has no impact on the service.

Problem is being tracked here and here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.