cloudflare / cloudflare/ebpf_exporter
Cgroup decoder shows `unknown_cgroup_id` even when the path exists (e.g. under controller mounts)
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 282
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 8
Description
Hi — I’ve been playing with ebpf_exporter and toying with the oomkill example in a small k8s cluster, and ran into something I thought was worth reporting.
In my setup, the exporter runs in Kubernetes with the host’s `/sys/fs/cgroup` explicitly mounted as a hostPath volume (readonly) so the pod sees the same cgroup tree as the host. Node is Ubuntu 20.04, kernel 5.15. Running the oomkill example, I get `cgroup_path="unknown_cgroup_id:12345"` in the metrics even though the cgroup is there. On the host, `find /sys/fs/cgroup -inum 12345` shows it under e.g. `/sys/fs/cgroup/memory/kubepods.slice/...`. So the inode is valid, and the path exists; the exporter just never adds it to its mapping.
From poking at the code: when fanotify is used, we only mark the single path `/sys/fs/cgroup`. That marks the tmpfs at the root. But the actual pod cgroups live under the *controller* mounts (memory, systemd, cpu, etc.), which are separate filesystems. So we never get FAN_CREATE/FAN_DELETE for those—the kernel only sends events for the fs we marked.
**Limiting factor:** `FAN_MARK_FILESYSTEM` applies to exactly one filesystem (the one containing the path we pass). We pass `/sys/fs/cgroup`, so we only get events for the tmpfs. Create/delete on any *other* mount (e.g. the memory cgroup fs) are not delivered to our mark.
Example `mount | grep cgroup` on a typical node:
```
tmpfs on /sys/fs/cgroup type tmpfs (...)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (...)
cgroup on /sys/fs/cgroup/systemd type cgroup (...,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (...,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (...,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (...)
...
```
So we mark the first line; OOM cgroup IDs resolve to paths under `memory` (second cgroup mount). Fanotify never sees those, and with fanotify there’s no re-walk on miss, so the id stays unresolved. One potential fix that might work: read `/proc/self/mountinfo`, find all cgroup-related mounts under the root, and call fanotify mark on each of them so we get events from every hierarchy.
Happy to try a PR in that direction for this repo if you think it is okay.
Contributor guide
Research direction
Start at the fanotify-based cgroup path mapping used by the oomkill example and trace how /sys/fs/cgroup is marked. Inspect /proc/self/mountinfo to identify the controller mounts that need coverage. Done means cgroup IDs created under those mounts resolve to paths instead of unknown_cgroup_id; verify with the reported Kubernetes setup and oomkill metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- observability, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100