cAdvisor v0.53.0 fails to collect container metrics on Docker 29.x with overlayfs storage driver (Rocky Linux 9.6)
- Dominant language
- Go
- Stars
- 19.4k
- Forks
- 2.5k
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 1
Description
## Description
cAdvisor v0.53.0 works correctly on Docker 28.5.x but fails to collect
container-level metrics on Docker 29.3.0 on Rocky Linux 9.6.
The issue appears to be related to Docker 29.x defaulting to `overlayfs`
storage driver instead of `overlay2` on RHEL-based systems.
## Environment
| Component | Working | Not Working |
|---|---|---|
| cAdvisor Version | v0.53.0 | v0.53.0 |
| Docker Version | 28.5.1 / 28.5.2 | 29.3.0 |
| Docker Storage Driver | overlay2 | overlayfs |
| OS | Rocky Linux 9.6 (Blue Onyx) | Rocky Linux 9.6 (Blue Onyx) |
| Kernel | 5.14.0-570.55.1.el9_6.x86_64 | 5.14.0-570.26.1.el9_6.x86_64 |
| Architecture | x86_64 | x86_64 |
| Cgroup Driver | systemd | systemd |
| Cgroup Version | v2 | v2 |
## Docker Info (Not Working - Docker 29.3.0)
```
Server Version: 29.3.0
Storage Driver: overlayfs
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Cgroup Driver: systemd
Cgroup Version: 2
Operating System: Rocky Linux 9.6 (Blue Onyx)
Kernel Version: 5.14.0-570.26.1.el9_6.x86_64
```
## Docker Info (Working - Docker 28.5.1)
```
Server Version: 28.5.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Cgroup Driver: systemd
Cgroup Version: 2
Operating System: Rocky Linux 9.6 (Blue Onyx)
Kernel Version: 5.14.0-570.55.1.el9_6.x86_64
```
## cAdvisor Compose Config
```yaml
cadvisor:
image: ghcr.io/google/cadvisor:v0.53.0
container_name: cadvisor
restart: unless-stopped
privileged: true
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /sys/fs/cgroup:/sys/fs/cgroup:ro
- /var/lib/docker:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
devices:
- /dev/kmsg
command:
- --housekeeping_interval=10s
```
## Error Logs
```
E0327 10:23:43.554710 1 manager.go:1119] Failed to create existing
container: /system.slice/docker-4d4a37941dc5e43ae2ed2a4faf841dc7f7b28de65a052c228576c39821923a73.scope:
failed to identify the read-write layer ID for container
"4d4a37941dc5e43ae2ed2a4faf841dc7f7b28de65a052c228576c39821923a73". -
open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/4d4a37941dc5e43ae2ed2a4faf841dc7f7b28de65a052c228576c39821923a73/mount-id:
no such file or directory
```
cAdvisor is trying to resolve container layer IDs under:
```
/rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/
```
instead of the standard:
```
/rootfs/var/lib/docker/image/overlay2/layerdb/mounts/
```
## Impact
- `/metrics` endpoint returns **only root cgroup** (`id="/"`) — no
individual container metrics
- Verification:
```bash
curl -s http://localhost:8080/metrics | grep container_memory_working_set_bytes | grep -v '^#'
# Returns only:
container_memory_working_set_bytes{id="/"} 2.94242304e+09
# Expected: individual container entries per running container
```
## Root Cause Analysis
Docker 29.x on RHEL/Rocky Linux 9.x defaults to `overlayfs`
(containerd snapshotter path convention) instead of the classic `overlay2`
graphdriver. cAdvisor v0.53.0 does not handle the `overlayfs` storage
driver path (`/var/lib/docker/image/overlayfs/`) when resolving container
layer IDs during startup recovery (`manager.go:1119`).
Additionally, `--docker_only=true` flag does **not** suppress these errors
during the startup recovery scan — only during ongoing housekeeping cycles.
## Workaround
Forcing `overlay2` in `/etc/docker/daemon.json` resolves the issue:
```json
{
"storage-driver": "overlay2"
}
```
However this requires wiping `/var/lib/docker` and redeploying all
containers, which is disruptive in production environments.
## Expected Behavior
cAdvisor should either:
1. Support `overlayfs` storage driver path used by Docker 29.x on
RHEL/Rocky Linux 9.x, **or**
2. Gracefully skip stale systemd cgroup scopes that cannot be resolved
instead of failing container metric collection entirely, **or**
3. Document that Docker 29.x with `overlayfs` driver is not supported
and requires explicit `overlay2` configuration
## Steps to Reproduce
1. Fresh Rocky Linux 9.6 VM
2. Install Docker Engine 29.3.0 (official Docker CE repo)
3. Verify storage driver: `docker info | grep "Storage Driver"` → `overlayfs`
4. Run cAdvisor v0.53.0 with standard compose config above
5. Check metrics: `curl -s http://localhost:8080/metrics | grep container_memory_working_set_bytes`
6. Observe only root cgroup metric returned, no individual container metrics
Contributor guide
Assessment
This issue has not been assessed yet.