nestybox / nestybox/sysbox

docker stats displays wrong MEM USAGE and MEM % only on sysbox containers

Open
#714 5 comments 1 reaction 1 assignee View on GitHub

@ctalledo is already working on this.

Since Oct 15, 2023.

bug
Dominant language
Shell
Stars
3.9k
Forks
230
Avg merge
7h 48m
Merged PRs (30d)
3

Description

The docker statscommand seems fundamentally flawed on sysbox containers, not showing changes in either cpu and memory usage, always stays on 1 PID, also not catching the limit when running with --memory constraint.

The issue https://github.com/nestybox/sysbox/issues/303 marked this as solved, but I found this problem still happening on my tests.

Steps

  1. First, Run two containers, one with sysbox runtime and another on runc:
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker run --privileged --name runc-container -d -P nestybox/ubuntu-bionic-systemd-docker:latest
62c382e8b2e194f6d8d2f65cdc578888abb0f12caffc73527bc94266b1d70439
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker run --runtime=sysbox-runc --name sysbox-container -d -P nestybox/ubuntu-bionic-systemd-docker:latest
36f226437bedabde221a7596846774a866fb089277478086dee22543963675c7
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker ps
CONTAINER ID   IMAGE                                          COMMAND                  CREATED          STATUS          PORTS                                     NAMES
36f226437bed   nestybox/ubuntu-bionic-systemd-docker:latest   "/sbin/init --log-le…"   3 seconds ago    Up 1 second     0.0.0.0:32770->22/tcp, :::32770->22/tcp   sysbox-container
62c382e8b2e1   nestybox/ubuntu-bionic-systemd-docker:latest   "/sbin/init --log-le…"   18 seconds ago   Up 17 seconds   0.0.0.0:32769->22/tcp, :::32769->22/tcp   runc-container
  1. then run docker stats to see memory and cpu usage:
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker stats --no-stream
CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
36f226437bed   sysbox-container   0.00%     2.277MiB / 3.808GiB   0.06%     3.41kB / 0B     0B / 8.19kB      1
62c382e8b2e1   runc-container     0.33%     77.99MiB / 3.808GiB   2.00%     7.12kB / 670B   14.4MB / 311kB   28

Look how the sysbox container wrongly show only 2.2M of memory usage, and there is no change on that, the container could be using Gigabytes of RAM but still a few Mb is shown.

the problem doesn't affect the 'normal' runc containers, showing correct ammounts of memory usage.

The system info on sys/fs/cgroup does show the correct ammounts:

erick@erick-VirtualBox:~/testedocker/docker-vm$ cat /sys/fs/cgroup/system.slice/docker-36f226437bedabde221a7596846774a866fb089277478086dee22543963675c7.scope/memory.current
64208896

System info

The test was run in a ubuntu virtual machine, with the following specs:

erick@erick-VirtualBox:~/testedocker/docker-vm$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 23.04
Release:	23.04
Codename:	lunar
erick@erick-VirtualBox:~/testedocker/docker-vm$ uname -a
Linux erick-VirtualBox 6.2.0-20-generic #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
erick@erick-VirtualBox:~/testedocker/docker-vm$ docker -v
Docker version 24.0.2, build cb74dfc
erick@erick-VirtualBox:~/testedocker/docker-vm$ sysbox-runc --version
sysbox-runc
	edition: 	Community Edition (CE)
	version: 	0.6.2
	commit: 	60ca93c783b19c63581e34aa183421ce0b9b26b7
	built at: 	Mon Jun 12 03:49:19 UTC 2023
	built by: 	Cesar Talledo
	oci-specs: 	1.0.2-dev

As a side question, what is the recommended OS version and kernel version for running sysbox runtime?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.