Mirantis / Mirantis/cri-dockerd
No workarounds for logging spam "docker does not support reopening container log files", when not using json-file logging driver
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 356
- Avg merge
- 4d 4h
- Merged PRs (30d)
- 3
Description
Expected Behavior
Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.509505 35502 container_log_manager.go:253] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files"
Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.512587 35502 remote_runtime.go:751] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files"
The SPAM is real.... for a 10 node cluster we had over 200,000 log lines of this nature in 1 hour.
There are no "workarounds" for users who do not use the 'json-file' logging driver.
Suggested workaround, from https://github.com/Mirantis/cri-dockerd/issues/35
fixed when setting container-log-max-size to the same size as docker daemon max-size log opts.
Are users just subject to the pain?
I saw in one thread that using a stub logging manager with cri-dockerd could work. Is there any documentation to help implement this?
This is our daemon configuration.
{
"max-concurrent-downloads": 6,
"exec-opts": [
"native.cgroupdriver=systemd"
],
"storage-driver": "overlay2",
"live-restore": true,
"log-driver": "fluentd",
"log-opts": {
...
}
}
Specifications
- Version: Kubernetes 1.29.2
- Platform: v0.3.11/cri-dockerd_0.3.11.3-0.ubuntu-jammy_amd64.deb
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the reported cri-dockerd behavior with Docker's fluentd logging driver and the referenced stub logging manager workaround. Compare the daemon.json logging configuration with Kubernetes 1.29.2 and cri-dockerd 0.3.11; done should be a documented workaround or a change that stops the repeated reopen-log errors for non-json-file drivers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100