Mirantis / Mirantis/cri-dockerd

Linux container stats report total memory usage as WorkingSetBytes

Open
#574 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.4k
Forks
356
Avg merge
4d 4h
Merged PRs (30d)
3

Description

Hello, I noticed a discrepancy between what cAdvisor reports and what the kubelet /stats/summary endpoint reports, pertaining to memory (working_set_bytes), when using cri-dockerd.

I found that cri-dockerd is seemingly reporting total usage (dockerStats.MemoryStats.Usage) as CRI's WorkingSetBytes. On Linux, this includes inactive file cache and differs from the working-set bytes reported by the Docker CLI, cAdvisor, and containerd.

And in particular for my usecase, it causes the kubelet /stats/summary endpoint to report a higher working-set memory for docker-backed nodes.

I believe the necessary keys exist already in .Stats to subtract the inactive file cache (total_inactive_file on cgroup v1, inactive_file on cgroup v2).

I am happy to provide any further information if necessary!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how dockerStats.MemoryStats.Usage is mapped to CRI WorkingSetBytes, then compare the cgroup v1 total_inactive_file and cgroup v2 inactive_file values mentioned in the report. Done means Linux working-set values exclude inactive file cache and the kubelet /stats/summary result aligns with Docker CLI, cAdvisor, and containerd.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.