Mirantis / Mirantis/cri-dockerd

No workarounds for logging spam "docker does not support reopening container log files", when not using json-file logging driver

Open
#337 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.4k
Forks
356
Avg merge
4d 4h
Merged PRs (30d)
3

Description

Expected Behavior

Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.509505   35502 container_log_manager.go:253] "Container log doesn't exist, reopen container log failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" 

Mar 18 14:54:14 kubelet[35502]: E0318 14:54:14.512587   35502 remote_runtime.go:751] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" 

The SPAM is real.... for a 10 node cluster we had over 200,000 log lines of this nature in 1 hour.
There are no "workarounds" for users who do not use the 'json-file' logging driver.
Suggested workaround, from https://github.com/Mirantis/cri-dockerd/issues/35

fixed when setting container-log-max-size to the same size as docker daemon max-size log opts.

Are users just subject to the pain?
I saw in one thread that using a stub logging manager with cri-dockerd could work. Is there any documentation to help implement this?

This is our daemon configuration.

{
    "max-concurrent-downloads": 6,
    "exec-opts": [
        "native.cgroupdriver=systemd"
    ],
    "storage-driver": "overlay2",
    "live-restore": true,
    "log-driver": "fluentd",
    "log-opts": {
        ...
    }
}

Specifications

  • Version: Kubernetes 1.29.2
  • Platform: v0.3.11/cri-dockerd_0.3.11.3-0.ubuntu-jammy_amd64.deb

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the reported cri-dockerd behavior with Docker's fluentd logging driver and the referenced stub logging manager workaround. Compare the daemon.json logging configuration with Kubernetes 1.29.2 and cri-dockerd 0.3.11; done should be a documented workaround or a change that stops the repeated reopen-log errors for non-json-file drivers.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.