Mirantis / Mirantis/cri-dockerd
kubelet: Unable to fetch container log stats: failed to get fsstats when container restart
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 356
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 3
Description
What happened?
Kebelet continues to display "Unable to fetch container log stats" errors trying to access missing logs of docker containers.
I'm not sure what is the mechanism here, while it seems that if somehow the container is not recycled by kebelet, the dangling link to the deleted container is not handled, triggering the file not found error.
cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_kube-scheduler-xxx_4xxxe/kube-scheduler/15.log\": no such file or directory" containerName="kube-scheduler"
cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_kube-apiserver-xxx_xxx/kube-apiserver/29.log\": no such file or directory" containerName="kube-apiserver"
cri_stats_provider.go:694] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_etcd-xxx_0x9/etcd/18.log\": no such file or directory" containerName="etcd"
What did you expect to happen?
automatically remove missing logs
How can we reproduce it (as minimally and precisely as possible)?
Started with kubeadm init --cri-socket unix:///var/run/cri-dockerd.sock.
Remove any exited container from docker
Then "Unable to fetch container log stats" error will be triggered
xref: https://github.com/kubernetes/kubernetes/issues/123231
version:
cri-dockerd v0.3.16
k8s v1.31.1
docker 1.27.x
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with cri_stats_provider.go around line 694 and reproduce the error after kubeadm init --cri-socket unix:///var/run/cri-dockerd.sock, then removing an exited Docker container. Trace how missing container log paths are handled; done means removed logs no longer trigger repeated "Unable to fetch container log stats" errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100