docker stats displays wrong MEM USAGE and MEM % only on sysbox containers
@ctalledo is already working on this.
Since Oct 15, 2023.
- Dominant language
- Shell
- Stars
- 3.9k
- Forks
- 230
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 3
Description
The docker statscommand seems fundamentally flawed on sysbox containers, not showing changes in either cpu and memory usage, always stays on 1 PID, also not catching the limit when running with --memory constraint.
The issue https://github.com/nestybox/sysbox/issues/303 marked this as solved, but I found this problem still happening on my tests.
Steps
- First, Run two containers, one with sysbox runtime and another on runc:
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker run --privileged --name runc-container -d -P nestybox/ubuntu-bionic-systemd-docker:latest
62c382e8b2e194f6d8d2f65cdc578888abb0f12caffc73527bc94266b1d70439
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker run --runtime=sysbox-runc --name sysbox-container -d -P nestybox/ubuntu-bionic-systemd-docker:latest
36f226437bedabde221a7596846774a866fb089277478086dee22543963675c7
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
36f226437bed nestybox/ubuntu-bionic-systemd-docker:latest "/sbin/init --log-le…" 3 seconds ago Up 1 second 0.0.0.0:32770->22/tcp, :::32770->22/tcp sysbox-container
62c382e8b2e1 nestybox/ubuntu-bionic-systemd-docker:latest "/sbin/init --log-le…" 18 seconds ago Up 17 seconds 0.0.0.0:32769->22/tcp, :::32769->22/tcp runc-container
- then run docker stats to see memory and cpu usage:
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
36f226437bed sysbox-container 0.00% 2.277MiB / 3.808GiB 0.06% 3.41kB / 0B 0B / 8.19kB 1
62c382e8b2e1 runc-container 0.33% 77.99MiB / 3.808GiB 2.00% 7.12kB / 670B 14.4MB / 311kB 28
Look how the sysbox container wrongly show only 2.2M of memory usage, and there is no change on that, the container could be using Gigabytes of RAM but still a few Mb is shown.
the problem doesn't affect the 'normal' runc containers, showing correct ammounts of memory usage.
The system info on sys/fs/cgroup does show the correct ammounts:
erick@erick-VirtualBox:~/testedocker/docker-vm$ cat /sys/fs/cgroup/system.slice/docker-36f226437bedabde221a7596846774a866fb089277478086dee22543963675c7.scope/memory.current
64208896
System info
The test was run in a ubuntu virtual machine, with the following specs:
erick@erick-VirtualBox:~/testedocker/docker-vm$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 23.04
Release: 23.04
Codename: lunar
erick@erick-VirtualBox:~/testedocker/docker-vm$ uname -a
Linux erick-VirtualBox 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker -v
Docker version 24.0.2, build cb74dfc
erick@erick-VirtualBox:~/testedocker/docker-vm$ sysbox-runc --version
sysbox-runc
edition: Community Edition (CE)
version: 0.6.2
commit: 60ca93c783b19c63581e34aa183421ce0b9b26b7
built at: Mon Jun 12 03:49:19 UTC 2023
built by: Cesar Talledo
oci-specs: 1.0.2-dev
As a side question, what is the recommended OS version and kernel version for running sysbox runtime?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.