Leaked container in metrics: race in create/destroy leaves orphaned alias
- Dominant language
- Go
- Stars
- 19.4k
- Forks
- 2.5k
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 1
Description
## What happened
A short-live container(docker/7f11f53b8c807b02c...) keeps being reported on `/metrics`, even long after its directory has been removed.
```
# Canonical key is gone:
$ curl -s "localhost:8080/api/v2.0/stats/docker/7f11f53b8c807b02c0c9ec622de063f105df739a9080f38f4f87dc44c572ad68?type=name&count=1"
unknown container "/docker/7f11f53b8c807b02c0c9ec622de063f105df739a9080f38f4f87dc44c572ad68"
```
```
# Alias key still alive
$ curl -s "localhost:8080/api/v2.0/stats/cool_brown?type=docker&count=1"
{"/docker/7f11f53b8c807b02c0c9ec622de063f105df739a9080f38f4f87dc44c572ad68":[{"timestamp":"2026-08-11T07:37:28.54208249Z", ...}]}
```
```
# cgroup directory is already deleted
$ find /sys/fs/cgroup -name '*7f11f53b...*'
(no output)
```
## Relevant codes
```go
// cadvisor/lib/manager/manager.go - createContainer
// Add the container name and all its aliases. The aliases must be within the namespace of the factory.
m.containers.Store(namespacedName, cont)
for _, alias := range cont.info.Aliases {
m.containers.Store(namespacedContainerName{
Namespace: cont.info.Namespace,
Name: alias,
}, cont)
}
```
```go
// cadvisor/lib/manager/manager.go - destroyContainer
// Remove the container from our records (and all its aliases).
m.containers.Delete(namespacedName)
for _, alias := range cont.info.Aliases {
m.containers.Delete(namespacedContainerName{
Namespace: cont.info.Namespace,
Name: alias,
})
}
```
`createContainer` / `destroyContainer` may operate the above code concurrently.
```
Timestamp 1: Routine A: Store name NamespacedKey
Timestamp 2: Routine B: Delete name NamespacedKey
Timestamp 3: Routine B: Delete name AliasKey
Timestamp 4: Routine A: Store name AliasKey
```
It leaves the **namespacedName key deleted but the alias keys orphaned**. Because reaping (`getContainersDiff`) and the destroy entry point both key off the canonical name, the orphan is never collectible — but `getSubcontainers` (used by `/metrics`) iterates *all* keys, so it keeps being exported forever.
## What you expected to happen
Once a container's cgroup is gone, all of its metrics stop being reported, and the entry is removed from the internal containerMap.
## Reproduction
Hard to reproduce deterministically (it's a timing race).
Contributor guide
Research direction
Start in cadvisor/lib/manager/manager.go by reading createContainer, destroyContainer, getContainersDiff, and getSubcontainers to trace how canonical and alias keys are managed. Reproduce or model the create/destroy interleaving described in the issue; done means a removed container leaves no alias entries and is no longer exported by /metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100