containers / containers/conmon

Max log size not applied when container restarts

Open
#266 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
499
Forks
150
Avg merge
1d 10h
Merged PRs (30d)
20

Description

The max log size option is set to 200MB through `log_size_max = 209715200` in /etc/containers/containers.conf (no other option is uncommented in it), but the container logs exceed 200MB. This seems to happen when a container restarts and thus the conmon process is restarted. In my case, podman restarts the container every few minutes and the bytes written by the previous conmon process do not seem to be taken in count when applying the max log size.

**Versions**
`CentOS 7`

`conmon --version`
`conmon version 2.0.21
commit: efe288263ebb1bab4673d2cfbe1a352f3c7d27c1-dirty`

`runc --version`
`runc version 1.0.0-rc10
commit: 4791ca3d31174576ddd8f4ff562283339196efa9
spec: 1.0.1-dev`

`rpm -qa | grep containers`
`containers-common-1.2.0-3.el7.x86_64`

`podman version`
`Version: 2.1.1
API Version: 2.0.0
Go Version: go1.15.2
Built: Wed Oct 28 07:37:02 2020
OS/Arch: linux/amd64`

**Results obtained**
This container's log file is over 800MB:
`ls -lah /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log`
`-rw-------. 1 root root 839M Jun 1 14:05 /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log`

I can see that the --log-size-max option is applied with the right value by conmon.
` ps -ef | grep conmon | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log
root 23953 1 0 14:04 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -u 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata -p /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/pidfile -n fluent-sd-pod-fluent-sd --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -s -l k8s-file:/var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log --log-size-max 209715200 --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7`

Container has been restarted recently:
`date`
`Tue Jun 1 14:36:06 CDT 2021`

`ps -eo pid,lstart,cmd | grep /var/lib/containers/storage/overlay-containers/0df62dc3df3784312e28227fa8937e143ad3ce1b87ea112af7b99ac08d23a9e7/userdata/ctr.log`
`17386 Tue Jun 1 14:32:21 2021 <...>`

**Results expected**
Container log file size not to exceed 200MB.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the restart scenario with /etc/containers/containers.conf, conmon --log-size-max 209715200, and the container's userdata/ctr.log. Compare log handling before and after conmon restarts; done means the log file remains at or below 200MB across container restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.