google / google/cadvisor

container_spec_cpu_quota not exported for main container when pod uses shareProcessNamespace: true

Open
#3,893 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.4k
Forks
2.5k
Avg merge
9h 7m
Merged PRs (30d)
1

Description

## Summary

When a Kubernetes pod is configured with `shareProcessNamespace: true`,
`container_spec_cpu_quota` is silently not exported for the **main application
container** in Prometheus. Only sidecar containers that are not the PID namespace
owner retain the metric. This causes all CPU utilization percentage calculations
that rely on this metric to be either completely absent or severely inflated (~60×).

---

## Environment

- **Kubernetes**: GKE, control plane v1.34.8-gke.1000000, nodes v1.34.6-gke.1307000 (confirmed); likely any Kubernetes cluster using containerd
- **Container runtime**: containerd v2.1.5
- **Node OS**: Container-Optimized OS from Google, kernel 6.12.68+
- **cAdvisor**: bundled with kubelet on nodes — confirmed affected across multiple GKE minor versions up to **v0.57.0** (latest at time of writing)
- **cgroup version**: v2 (COS default on GKE since kernel 5.x)

---

## Minimal Reproduction

Deploy a pod with `shareProcessNamespace: true` and a sidecar:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: test-shared-pid
spec:
shareProcessNamespace: true
containers:
- name: main-app
image: nginx
resources:
requests:
cpu: "1"
limits:
cpu: "1"
- name: sidecar
image: busybox
command: ["sleep", "infinity"]
resources:
requests:
cpu: 100m
limits:
cpu: 100m
```

Then query Prometheus:

```promql
container_spec_cpu_quota{pod="test-shared-pid"}
```

---

## Expected Behavior

Two series present — one per container:

```
container_spec_cpu_quota{container="main-app", pod="test-shared-pid"} = 100000
container_spec_cpu_quota{container="sidecar", pod="test-shared-pid"} = 10000
```

---

## Actual Behavior

Only the **sidecar's** quota is exported:

```
container_spec_cpu_quota{container="sidecar", pod="test-shared-pid"} = 10000
```

`container_spec_cpu_quota` for `main-app` is completely absent. The main container
has a CPU limit set and the limit IS enforced by the kernel — the cgroup
`cpu.cfs_quota_us` file contains the correct value — but cAdvisor does not export it.

---

## What Is and Isn't Affected

| Metric | Affected? |
|---|---|
| `container_spec_cpu_quota` | ✅ Yes — absent for main container |
| `container_spec_cpu_period` | ✅ Yes — absent for main container |
| `container_cpu_usage_seconds_total` | ❌ No — correctly exported for all containers |
| `container_memory_working_set_bytes` | ❌ No — correctly exported for all containers |
| `container_spec_memory_limit_bytes` | ❌ No — correctly exported for all containers |

Only **OCI-spec-derived** metrics are affected. cAdvisor reads CPU _usage_ from
cgroup stat counters (unaffected), but reads CPU _quota_ via a path that is
disrupted by `shareProcessNamespace`.

Removing `shareProcessNamespace: true` from the pod spec immediately restores
`container_spec_cpu_quota` for the main container.

---

## Impact

Adding a sidecar that requires `shareProcessNamespace: true` to read `/proc` entries
of sibling containers causes a widely-used PromQL CPU utilization pattern to
produce severely inflated values (~60×):

```promql
sum(rate(container_cpu_usage_seconds_total{pod=~"$pod"}[5m])) by (pod)
/
sum(container_spec_cpu_quota{pod=~"$pod"} / container_spec_cpu_period{pod=~"$pod"}) by (pod)
```

**Before** (no sidecar, no `shareProcessNamespace`):
- denominator = 600,000 µs = 6 cores ✅

**After** (sidecar added, `shareProcessNamespace: true`):
- `container_spec_cpu_quota{container="main-app"}` → **absent**
- `container_spec_cpu_quota{container="sidecar"}` → 10,000 µs = 100m
- denominator collapses to 10,000 µs = **0.1 cores** → CPU % inflated ~60×

Alert rules that include an explicit `container=` label filter in the denominator
go **silently dark** (no data → no alert, even at genuine high CPU), which is
arguably more dangerous than the inflation.

---

## Root Cause Analysis

### How `container_spec_cpu_quota` is built

The metric is populated in `container/common/helpers.go::getSpecInternal()` by
reading directly from the **cgroup filesystem**:

```go
// cgroup v1
quota := readString(cpuRoot, "cpu.cfs_quota_us")
if quota != "" && quota != "-1" {
val, err := strconv.ParseUint(quota, 10, 64)
if err == nil {
spec.Cpu.Quota = val
}
}
```

```go
// cgroup v2
max := readString(cpuRoot, "cpu.max")
if splits[0] != "max" {
spec.Cpu.Quota = parseUint64String(splits[0])
}
```

`Quota` stays `0` when the cgroup reports `-1` (unlimited). The containerd handler
in `container/containerd/handler.go::GetSpec()` then calls:

```go
spec, err := common.GetSpec(h.cgroupPaths, h.machineInfoFactory, hasNet, hasFilesystem)
```

And `metrics/prometheus.go` only exports the metric when `Quota != 0`:

```go
if cont.Spec.Cpu.Quota != 0 {
ch <- prometheus.MustNewConstMetric("container_spec_cpu_quota", ...)
}
```

There is no explicit suppression — the metric simply never gets set because
the wrong cgroup is being read, which reports `-1`.

### Why the wrong cgroup is read

When `shareProcessNamespace: true` is set, containerd designates the
infra/pause container as the **PID namespace owner**. The main application container
is created with a shared PID namespace reference in its OCI spec:

```json
"linux": {
"namespaces": [
{ "type": "pid", "path": "/proc//ns/pid" }
],
"cgroupsPath": "/kubepods/guaranteed/pod/"
}
```

The `cgroupsPath` field correctly points to the container's own cgroup. However,
cAdvisor's `cgroupPaths` initialization in `newContainerdContainerHandler()` — when
it encounters the shared PID namespace entry — appears to resolve the container's
cgroup to the **pause/infra container's cgroup** rather than the container's own.

The pause container has no CPU limit → its cgroup reports `cpu.cfs_quota_us = -1`
→ `getSpecInternal()` skips setting `Quota` → `Quota` stays `0` → metric not
exported.

The sidecar is also in the shared PID namespace but is **not the namespace owner**,
so it is processed differently and its own cgroup (with the correct limit) is read.

---

## Suggested Fix

### Where to look

**`container/containerd/handler.go` — `newContainerdContainerHandler()`**

Inspect how `h.cgroupPaths` is populated. Verify the cgroup path is always derived
from the **container's own** `spec.Linux.CgroupsPath` OCI field, and never
influenced by the PID namespace owner reference at
`spec.Linux.Namespaces[type=pid].path`.

### Proposed fix direction

Ensure `cgroupPaths` is always derived from the container's own `linux.cgroupsPath`
OCI spec field, regardless of namespace sharing:

```go
// Always use the container's own cgroup — do NOT follow the PID namespace
// owner reference to resolve the cgroup path
if spec.Linux != nil && spec.Linux.CgroupsPath != "" {
cgroupPath = spec.Linux.CgroupsPath
}
```

The fix cannot be in `getSpecInternal()` — it only receives the resolved cgroup path
and has no knowledge of namespace topology. The fix must be upstream at cgroup path
resolution time in the containerd handler.

### Verification

After the fix, querying with `shareProcessNamespace: true` should yield both series:

```promql
container_spec_cpu_quota{container="main-app"} =
container_spec_cpu_quota{container="sidecar"} =
```

---

## Workaround

Replace `container_spec_cpu_quota / container_spec_cpu_period` with
`kube_pod_container_resource_limits{resource="cpu"}` from **kube-state-metrics**,
which reads from the Kubernetes API and is completely unaffected by this bug:

```promql
# Before (broken when shareProcessNamespace: true)
rate(container_cpu_usage_seconds_total{container="main-app"}[5m])
/ (container_spec_cpu_quota{container="main-app"} / container_spec_cpu_period{container="main-app"})

# After (correct regardless of namespace configuration)
rate(container_cpu_usage_seconds_total{container="main-app"}[5m])
/ kube_pod_container_resource_limits{container="main-app", resource="cpu"}
```

---

## Additional Notes

- The bug is **100% reproducible** by adding/removing `shareProcessNamespace: true`
with no other changes
- `container_cpu_usage_seconds_total` is **not affected** — cAdvisor reads CPU
usage from cgroup stat counters, not OCI spec paths; only `container_spec_*`
metrics are impacted
- No fix found in any cAdvisor release up to **v0.57.0**
- Related but distinct prior issues: #3154, #3601, #2220 (missing quota metrics,
not in the context of shared PID namespaces)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.