[bug] Ubuntu 24 → cgroups issue: app-level scrape via io.cadvisor.metric.prometheus-* label fails after container (re)starts
- Dominant language
- Go
- Stars
- 19.4k
- Forks
- 2.5k
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 1
Description
**Summary:**
Since upgrading hosts from **Ubuntu 20.04 to Ubuntu 24.04**, cAdvisor intermittently fails to register external Prometheus collectors configured via the io.cadvisor.metric.prometheus- container label. Core host/container metrics (CPU/RAM/etc.) keep working, but application-level scraping (HTTP /metrics) stops for all containers on the machine after a container starts/restarts. Restarting cAdvisor temporarily restores app-level scraping.
Using --cgroupns=host makes it work initially, but the failure reappears on the next container (re)start.
**How we configure per-container scraping**
We use cAdvisor’s feature to scrape app metrics from additional endpoints, keyed by a label on the target container:
- Label on the target container:
io.cadvisor.metric.prometheus-myapp=/var/cadvisor/cadvisor_config.json
- The JSON file cAdvisor reads:
{"endpoint": "http://169.244.0.1:19100/metrics"}
**What happens**
After a container (re)start, cAdvisor logs:
failed to register collectors for /system.slice/docker-.scope: failed to read config file "prometheus-myapp" for config /var/cadvisor/cadvisor_config.json container /system.slice/docker-.scope failed to execute "/usr/sbin/chroot" command exit status 1
From that point, app-level scraping stops for all containers on the host. Host/container resource metrics continue to be exported. Restarting the cAdvisor container restores app scraping until the next container (re)start.
**Steps to reproduce**
1. Host on Ubuntu 24.04 (worked on 20.04).
2. Run cAdvisor and mount the config path that the label references.
3. Run a target container with label:
io.cadvisor.metric.prometheus-myapp=/var/cadvisor/cadvisor_config.json
Ensure /var/cadvisor/cadvisor_config.json inside cAdvisor contains:
{"endpoint": "http://:/metrics"}
4. Observe that app metrics are scraped.
5. Restart the target container or start another container on the host.
6. Observe the error above; app-level scraping stops for all containers.
**Expected behavior**
cAdvisor should continue to read the label-referenced config and scrape the configured HTTP endpoint for every labeled container, across container (re)starts.
**Actual behavior**
After a container (re)start, cAdvisor logs a failed to execute "/usr/sbin/chroot" error for the systemd slice and stops app-level scraping globally until cAdvisor is restarted.
**Workaround tried**
Running cAdvisor with --cgroupns=host:
Works initially, but the failure recurs after the next container (re)start.
Restarting cAdvisor:
Temporarily restores app-level scraping.
**Environment**
Host OS: Ubuntu 24.04 (upgraded from 20.04)
Container runtime: Docker
cAdvisor : v0.52.1
cgroups: v2 on Ubuntu 24.04
cAdvisor flags: --docker_only=true --max_procs=1 --application_metrics_count_limit=1000 --storage_duration=30s
Contributor guide
Assessment
This issue has not been assessed yet.