[BUG] #3502: The container_fs_usage_bytes metric is too large, including the size of the container's read-only layer
- Dominant language
- Go
- Stars
- 19.4k
- Forks
- 2.5k
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 1
Description
Description:
When cadvisor supports collecting disk usage metrics of containerd containers, it uses the root directory of the container, causing the value of the container_fs_usage_bytes metric to include the size of the read-only layer.
// container/containerd/handler.go L152:156
`// Add the name and bare ID as aliases of the container.
handler.image = cntr.Image
if handler.includedMetrics.Has(container.DiskUsageMetrics) {
handler.rootfsStorageDir = path.Join(rootfsDir, *ArgContainerdNamespace, id, "rootfs")
handler.fsHandler = common.NewFsHandler(common.DefaultPeriod, handler.rootfsStorageDir, "", fsInfo)
}`
The rootfsStorageDir value constructed here is: /run/containerd/io.containerd.runtime.v2.task/k8s.io/{container id}/roofts, which is the container root directory. Then, GetDirUsage of fs.go is used to calculate the disk usage of its directory and subdirectories.
We all know that the container_fs_usage_bytes metric indicates the usage of the container's writable layer. Therefore, this value is inaccurate, and I have actually verified this result.
Steps to Reproduce:
1. Deploy this code independently
2. Query the container_fs_usage_bytes indicator of the container on the node
3. Log in to the node and view the writable layer directory of the container on the host
4. Found that the value of the container_fs_usage_bytes indicator > the size of the writable layer directory
Expected:
`handler.fsHandler = common.NewFsHandler(common.DefaultPeriod, handler.rootfsStorageDir, "", fsInfo)`
The rootfsStorageDir used should be the container's writable layer directory. Similar to docker factory, the writable layer directory is used when constructing fsHandler.
ENV:
- os: centos 7
- commit: 9e7a9392(from PR #3502 )
Related PR: #3502
Contributor guide
Assessment
This issue has not been assessed yet.