google / google/cadvisor

Leaked container in metrics: race in create/destroy leaves orphaned alias

Open
#3,924 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.4k
Forks
2.5k
Avg merge
9h 7m
Merged PRs (30d)
1

Description

## What happened
A short-live container(docker/7f11f53b8c807b02c...) keeps being reported on `/metrics`, even long after its directory has been removed.
```
# Canonical key is gone:
$ curl -s "localhost:8080/api/v2.0/stats/docker/7f11f53b8c807b02c0c9ec622de063f105df739a9080f38f4f87dc44c572ad68?type=name&count=1"
unknown container "/docker/7f11f53b8c807b02c0c9ec622de063f105df739a9080f38f4f87dc44c572ad68"
```
```
# Alias key still alive
$ curl -s "localhost:8080/api/v2.0/stats/cool_brown?type=docker&count=1"
{"/docker/7f11f53b8c807b02c0c9ec622de063f105df739a9080f38f4f87dc44c572ad68":[{"timestamp":"2026-08-11T07:37:28.54208249Z", ...}]}
```

```
# cgroup directory is already deleted
$ find /sys/fs/cgroup -name '*7f11f53b...*'
(no output)
```

## Relevant codes

```go
// cadvisor/lib/manager/manager.go - createContainer
// Add the container name and all its aliases. The aliases must be within the namespace of the factory.
m.containers.Store(namespacedName, cont)
for _, alias := range cont.info.Aliases {
m.containers.Store(namespacedContainerName{
Namespace: cont.info.Namespace,
Name: alias,
}, cont)
}
```
```go
// cadvisor/lib/manager/manager.go - destroyContainer
// Remove the container from our records (and all its aliases).
m.containers.Delete(namespacedName)
for _, alias := range cont.info.Aliases {
m.containers.Delete(namespacedContainerName{
Namespace: cont.info.Namespace,
Name: alias,
})
}
```

`createContainer` / `destroyContainer` may operate the above code concurrently.
```
Timestamp 1: Routine A: Store name NamespacedKey
Timestamp 2: Routine B: Delete name NamespacedKey
Timestamp 3: Routine B: Delete name AliasKey
Timestamp 4: Routine A: Store name AliasKey
```

It leaves the **namespacedName key deleted but the alias keys orphaned**. Because reaping (`getContainersDiff`) and the destroy entry point both key off the canonical name, the orphan is never collectible — but `getSubcontainers` (used by `/metrics`) iterates *all* keys, so it keeps being exported forever.

## What you expected to happen
Once a container's cgroup is gone, all of its metrics stop being reported, and the entry is removed from the internal containerMap.

## Reproduction
Hard to reproduce deterministically (it's a timing race).

Contributor guide

Open the contributing guide

Research direction

Start in cadvisor/lib/manager/manager.go by reading createContainer, destroyContainer, getContainersDiff, and getSubcontainers to trace how canonical and alias keys are managed. Reproduce or model the create/destroy interleaving described in the issue; done means a removed container leaves no alias entries and is no longer exported by /metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.