Mirantis / Mirantis/cri-dockerd

kubelet: Unable to fetch container log stats: failed to get fsstats when container restart

Open
#453 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.4k
Forks
356
Avg merge
4d 4h
Merged PRs (30d)
3

Description

What happened?

Kebelet continues to display "Unable to fetch container log stats" errors trying to access missing logs of docker containers.
I'm not sure what is the mechanism here, while it seems that if somehow the container is not recycled by kebelet, the dangling link to the deleted container is not handled, triggering the file not found error.

cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_kube-scheduler-xxx_4xxxe/kube-scheduler/15.log\": no such file or directory" containerName="kube-scheduler"
cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_kube-apiserver-xxx_xxx/kube-apiserver/29.log\": no such file or directory" containerName="kube-apiserver"
cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_etcd-xxx_0x9/etcd/18.log\": no such file or directory" containerName="etcd"
What did you expect to happen?

automatically remove missing logs

How can we reproduce it (as minimally and precisely as possible)?

Started with kubeadm init --cri-socket unix:///var/run/cri-dockerd.sock.
Remove any exited container from docker
Then "Unable to fetch container log stats" error will be triggered

xref: https://github.com/kubernetes/kubernetes/issues/123231

version:
cri-dockerd v0.3.16
k8s v1.31.1
docker 1.27.x

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cri_stats_provider.go around line 694 and reproduce the error after kubeadm init --cri-socket unix:///var/run/cri-dockerd.sock, then removing an exited Docker container. Trace how missing container log paths are handled; done means removed logs no longer trigger repeated "Unable to fetch container log stats" errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.