google / google/cadvisor

feat: expose metrics for stopped/exited containers with exit code and OOM detection

Open
#3,880 1 comment 2 reactions 0 assignees View on GitHub
lifecycle/stale
Dominant language
Go
Stars
19.4k
Forks
2.5k
Avg merge
9h 7m
Merged PRs (30d)
1

Description

## Problem

cAdvisor only exposes metrics for **running containers**. Once a container stops or exits, all its metrics disappear from Prometheus. There is no native way to:

- Monitor container state transitions (`running` → `exited`)
- Track exit codes via Prometheus labels
- Detect OOM kills after the fact (the native `container_oom_events_total` only works for running containers)
- Alert on containers that keep crashing and restarting

## Proposed solution

A new metric `docker_container_info` (constant gauge = 1) that queries the Docker/Moby socket for **all** containers (including stopped/exited), exposing:

```
docker_container_info{
container_id="abc123",
name="my-service",
image="nginx:latest",
state="exited",
exit_code="137 (Killed/OOM)"
} 1
```

Labels:
- `container_id` — full container ID
- `name` — container name
- `image` — image reference
- `state` — Docker state: `running`, `exited`, `paused`, etc.
- `exit_code` — human-readable exit code label, with OOM detection via `ContainerInspect.State.OOMKilled`

## Use cases

- **Alerting on crash loops**: `count_over_time(docker_container_info{state="exited"}[5m]) > 3`
- **OOM detection**: `docker_container_info{exit_code=~"137.*"}`
- **Inventory of all containers** (not just running ones) for auditing

## Implementation notes

This was implemented as an internal patch on a fork of cAdvisor (tested on v0.53.x and v0.57.0). It uses `ContainerList(All: true)` + `ContainerInspect` from the Moby API. The collector is registered alongside the existing Prometheus collectors in `RegisterPrometheusHandler`.

Since v0.56.0 dropped `github.com/docker/docker` in favour of `github.com/moby/moby`, a clean upstream implementation would use the moby client directly — possibly integrated into the existing Docker container factory rather than as a standalone handler-level collector.

Happy to submit a PR if the maintainers consider this useful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.