cloudflare / cloudflare/ebpf_exporter

Cgroup decoder shows `unknown_cgroup_id` even when the path exists (e.g. under controller mounts)

Open
#613 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.6k
Forks
282
Avg merge
5d 7h
Merged PRs (30d)
8

Description

Hi — I’ve been playing with ebpf_exporter and toying with the oomkill example in a small k8s cluster, and ran into something I thought was worth reporting.

In my setup, the exporter runs in Kubernetes with the host’s `/sys/fs/cgroup` explicitly mounted as a hostPath volume (readonly) so the pod sees the same cgroup tree as the host. Node is Ubuntu 20.04, kernel 5.15. Running the oomkill example, I get `cgroup_path="unknown_cgroup_id:12345"` in the metrics even though the cgroup is there. On the host, `find /sys/fs/cgroup -inum 12345` shows it under e.g. `/sys/fs/cgroup/memory/kubepods.slice/...`. So the inode is valid, and the path exists; the exporter just never adds it to its mapping.

From poking at the code: when fanotify is used, we only mark the single path `/sys/fs/cgroup`. That marks the tmpfs at the root. But the actual pod cgroups live under the *controller* mounts (memory, systemd, cpu, etc.), which are separate filesystems. So we never get FAN_CREATE/FAN_DELETE for those—the kernel only sends events for the fs we marked.

**Limiting factor:** `FAN_MARK_FILESYSTEM` applies to exactly one filesystem (the one containing the path we pass). We pass `/sys/fs/cgroup`, so we only get events for the tmpfs. Create/delete on any *other* mount (e.g. the memory cgroup fs) are not delivered to our mark.

Example `mount | grep cgroup` on a typical node:

```
tmpfs on /sys/fs/cgroup type tmpfs (...)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (...)
cgroup on /sys/fs/cgroup/systemd type cgroup (...,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (...,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (...,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (...)
...
```

So we mark the first line; OOM cgroup IDs resolve to paths under `memory` (second cgroup mount). Fanotify never sees those, and with fanotify there’s no re-walk on miss, so the id stays unresolved. One potential fix that might work: read `/proc/self/mountinfo`, find all cgroup-related mounts under the root, and call fanotify mark on each of them so we get events from every hierarchy.

Happy to try a PR in that direction for this repo if you think it is okay.

Contributor guide

Open the contributing guide

Research direction

Start at the fanotify-based cgroup path mapping used by the oomkill example and trace how /sys/fs/cgroup is marked. Inspect /proc/self/mountinfo to identify the controller mounts that need coverage. Done means cgroup IDs created under those mounts resolve to paths instead of unknown_cgroup_id; verify with the reported Kubernetes setup and oomkill metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
observability, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.