containerd / containerd/cgroups
cgroup v1 blkio parser aborts on "(unknown)" device rows, causing all container stats to fail
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 252
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
On cgroup v1 hosts with Linux native NVMe multipath enabled and multi-port / multi-controller NVMe SSDs, blkio cgroup files can contain rows where the device field is `(unknown)` instead of `major:minor`.
`containerd/cgroups` appears to treat that as a fatal parse error in the blkio stats parser. In Docker/containerd, the user-visible result is that `docker stats --no-stream` returns an all-zero stats row for the container, including unrelated metrics such as network, memory, and PIDs.
I think `containerd/cgroups` should tolerate these blkio rows by skipping them or preserving them as unknown-device blkio entries, rather than failing the entire container stats collection.
## Environment
Observed on multiple affected hosts:
- Docker Engine: `28.5.2`, `29.4.2`
- containerd: `1.7.29`, `2.2.3`
- runc: `1.3.3`, `1.3.5`
- Kernel: Ubuntu generic `6.8.0-111` and `6.8.0-117`
- Cgroups: v1 / cgroupfs
- `nvme_core.multipath=Y`
Upstream check:
- Current `containerd/cgroups` `main` was checked at commit https://github.com/containerd/cgroups/commit/360fd8b6c2ce402eddcbe6291e2393b0f202e253.
- The cgroup v1 blkio code currently lives under `cgroup1/blkio.go`.
- A local scratch unit test against that commit confirmed that a row beginning with `(unknown)` still returns `strconv.ParseUint: parsing "(unknown)": invalid syntax`.
Affected storage examples:
- `SNM5C-R3R8NC`, observed with `cmic=0x3`, hidden `nvme*c*n1` path disks
- Samsung `MZWLJ3T8HBLS` / PM1733-family drive, observed with `cmic=0x3`; Samsung PM1733 documentation describes [dual-port functionality](https://download.semiconductor.samsung.com/resources/brochure/PM1733%20NVMe%20SSD.pdf)
Healthy comparison host:
- Dell/Kioxia CD8 U.2 3.84TB, observed with `cmic=0x0`, no hidden NVMe path disks, and normal Docker stats; Kioxia documents CD8-R as a PCIe Gen4 x4 NVMe data-center SSD [here](https://americas.kioxia.com/en-us/business/ssd/data-center-ssd/cd8-r.html)
For CMIC meaning, libnvme documents `NVME_CTRL_CMIC_MULTI_CTRL` as indicating that the NVM subsystem may contain two or more controllers and may provide multiple paths for a single host: [Ubuntu manpage](https://manpages.ubuntu.com/manpages/noble/man2/nvme_id_ctrl_cmic.2.html).
## Actual Behavior
For affected running containers:
```text
$ docker stats --no-stream --format 'stats={{.Name}} net={{.NetIO}} block={{.BlockIO}} mem={{.MemUsage}} pids={{.PIDs}}'
stats= net=0B / 0B block=0B / 0B mem=0B / 0B pids=0
```
containerd logs show:
```text
strconv.ParseUint: parsing "(unknown)": invalid syntax
```
The container is not actually idle. Raw cgroup data, runc stats, and interface counters show nonzero activity. The stats failure appears to be triggered by blkio parsing.
Example blkio rows:
```text
(unknown) Read 0
(unknown) Write 0
(unknown) Sync 0
(unknown) Async 0
```
On one affected host, `blkio.throttle.io_serviced` had 24 `(unknown)` rows for the container, and related blkio files had the same pattern.
## Expected Behavior
A malformed or unknown blkio device row should not cause the whole container stats response to fail.
At minimum, `containerd/cgroups` should skip `(unknown)` blkio rows and still return CPU, memory, PIDs, network, and any parseable blkio metrics.
## Why This Looks Like a containerd/cgroups Parser Issue
The cgroup v1 blkio documentation describes these files as normally using `major:minor operation value` format: [Linux blkio controller docs](https://docs.kernel.org/admin-guide/cgroup-v1/blkio-controller.html).
The current blkio parser in `containerd/cgroups` splits each row on spaces and colons, then unconditionally parses `fields[0]` and `fields[1]` as numeric major/minor values: [`cgroup1/blkio.go`](https://github.com/containerd/cgroups/blob/360fd8b6c2ce402eddcbe6291e2393b0f202e253/cgroup1/blkio.go#L176-L220).
For a row like:
```text
(unknown) Read 0
```
that means:
```text
fields[0] = "(unknown)"
fields[1] = "Read"
fields[2] = "0"
```
So `strconv.ParseUint(fields[0], 10, 64)` fails and the parser returns an error.
### Minimal Reproducer Against Current Main
This test can be placed in package `cgroup1` against current `main`. It fails today with `strconv.ParseUint: parsing "(unknown)": invalid syntax`.
```go
package cgroup1
import (
"os"
"path/filepath"
"testing"
v1 "github.com/containerd/cgroups/v3/cgroup1/stats"
)
func TestReadEntryUnknownDeviceShouldNotFail(t *testing.T) {
root := t.TempDir()
cgroupPath := filepath.Join(root, "blkio", "test")
if err := os.MkdirAll(cgroupPath, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(
filepath.Join(cgroupPath, "blkio.throttle.io_serviced"),
[]byte("(unknown) Read 0\nTotal 0\n"),
0o644,
); err != nil {
t.Fatal(err)
}
ctrl := NewBlkio(root)
var entries []*v1.BlkIOEntry
if err := ctrl.readEntry(map[deviceKey]string{}, "test", "throttle.io_serviced", &entries); err != nil {
t.Fatalf("unknown blkio device rows should not fail all stats: %v", err)
}
}
```
## Why `(unknown)` Can Come From the Kernel
This seems to be triggered by Linux native NVMe multipath hidden path disks.
Linux NVMe multipath is documented as integrating namespaces with the same identifier into a single block device: [Linux NVMe multipath docs](https://docs.kernel.org/admin-guide/nvme-multipath.html).
The kernel source shows the `nvme_core` `multipath` module parameter: [`drivers/nvme/host/multipath.c`](https://codebrowser.dev/linux/linux/drivers/nvme/host/multipath.c.html). For multipath namespace paths, the NVMe driver names path disks like `nvme%dc%dn%d` and marks them hidden with `GENHD_FL_HIDDEN`: [`drivers/nvme/host/core.c`](https://codebrowser.dev/linux/linux/drivers/nvme/host/core.c.html#4150).
`GENHD_FL_HIDDEN` is documented in kernel source as a hidden block device used for underlying components of multipath devices: [`include/linux/blkdev.h`](https://codebrowser.dev/linux/linux/include/linux/blkdev.h.html#79).
Hidden disks skip normal BDI registration in `genhd.c`: [`block/genhd.c`](https://codebrowser.dev/linux/linux/block/genhd.c.html#538). The BDI fallback name is intentionally `(unknown)` when there is no BDI device: [`mm/backing-dev.c`](https://codebrowser.dev/linux/linux/mm/backing-dev.c.html#1219).
So the kernel appears able to produce `(unknown)` blkio device names for hidden multipath path devices. Even if that kernel behavior is imperfect, the container stats collector should not fail all metrics because of it.
## Workaround Validation: Disabling Native NVMe Multipath Removed the Bad Rows
On one validation host with the same hidden NVMe path disk signature, disabling Linux native NVMe multipath removed the `(unknown)` blkio rows and restored `docker stats` output.
Before the change:
- Docker Engine: `29.4.2`
- containerd: `2.2.3`
- Kernel: Ubuntu generic `6.8.0-111`
- Cgroups: v1 / cgroupfs
- `nvme_core.multipath=Y`
- Four hidden `nvme*c*n1` path disks were present for the data drives.
- blkio cgroup files contained `(unknown)` rows.
- containerd logs included `strconv.ParseUint: parsing "(unknown)": invalid syntax` during metrics collection.
The host was not using device-mapper multipath for the data volume. Its data filesystem was on an mdadm RAID10 array built directly from the NVMe namespace devices. The workaround was:
```bash
# added to GRUB_CMDLINE_LINUX_DEFAULT
nvme_core.multipath=N
sudo update-grub
sudo systemctl reboot
```
Docker and containerd were not upgraded as part of this test.
After reboot:
```text
/proc/cmdline included nvme_core.multipath=N
/sys/module/nvme_core/parameters/multipath = N
hidden nvme*c*n* path disks = none
mdadm RAID10 array = active, 4/4 members [UUUU]
/var/lib/docker = mounted on the same XFS filesystem
unknown blkio rows under /sys/fs/cgroup/blkio = 0
```
A disposable local smoke container then returned nonzero Docker stats:
```text
$ docker stats --no-stream --format 'name={{.Name}} net={{.NetIO}} block={{.BlockIO}} mem={{.MemUsage}} cpu={{.CPUPerc}}'
name= net=736B / 126B block=5.59MB / 0B mem=2.918MiB / 377.3GiB cpu=0.00%
```
There were no `(unknown)` blkio rows while that test container existed, and no new containerd metrics parse errors appeared.
This does not remove the need for a parser fix. Disabling native NVMe multipath is only safe for hosts that are not actually relying on NVMe multipath. The broader issue is still that one unknown blkio device row can make containerd/Docker lose all container stats.
## Related Reports
This looks closely related to [containerd/cgroups#300](https://github.com/containerd/cgroups/issues/300), which reports `strconv.ParseUint: parsing "(unknown)": invalid syntax`.
A downstream symptom also appears in the Kepler discussion [sustainable-computing-io/kepler#750](https://github.com/sustainable-computing-io/kepler/discussions/750), where metrics are reported as all zero and the linked containerd/cgroups issue is referenced.
## Possible Fix
In the cgroup v1 blkio parser, tolerate rows whose device field is not numeric `major:minor`: [`cgroup1/blkio.go`](https://github.com/containerd/cgroups/blob/360fd8b6c2ce402eddcbe6291e2393b0f202e253/cgroup1/blkio.go#L176-L220).
For example:
- Continue skipping `Total` as today.
- If `fields[0] == "(unknown)"`, skip the row or add an unknown-device blkio entry.
- More generally, if major/minor parse fails for a blkio stat row, do not fail the whole stats collection.
- Preserve the parse error only if strict parsing is explicitly desired.
Losing blkio accounting for an unknown hidden device is much less harmful than returning zero for unrelated container metrics such as network, memory, and PIDs.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.