google / google/cadvisor

Verified: 0e107a38 (fs/zfs VFS fallback) fixes a fatal kubelet crash on the ZFS graph driver, but is unreleased — request a release

Open
#3,922 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
19.4k
Forks
2.5k
Avg merge
9h 7m
Merged PRs (30d)
1

Description

## Summary

Commit 0e107a38 ("fs/zfs: fall back to VFS when zfs stats are unavailable",
part of the #3918 fix) is on `master` but is **not in any tagged release**
(`git tag --contains 0e107a38` is empty; it landed after v0.60.5). Because
kubelet vendors a released cadvisor, downstreams still hit a fatal kubelet
startup crash on ZFS-backed nodes. Requesting a release be cut so the fix is
consumable.

## Why it matters (the failure the fix closes)

On a Kubernetes node whose container store uses the **podman/containerd ZFS
graph driver** (e.g. a `kind` node run under podman with
`storage.driver = "zfs"`), `/dev/zfs` is present in the node but the `zfs` CLI
is **not installed**. cadvisor's `GetZfsStats` shells out to `zfs list ...`,
which fails with exec ENOENT (exit 127).

In released cadvisor (v0.56.2, as vendored by kubelet v1.36.1 — and unchanged
through v0.60.5), `fs/zfs/plugin.go GetStats` does:

```go
capacity, free, avail, err := getZfsStats(device)
if err != nil {
return nil, err // <-- no VFS fallback
}
```

So no `Fs` is produced for the ZFS rootfs partition, and the kubelet dies:

```
container_manager_linux.go "Unable to get rootfs data from cAdvisor interface"
err="cannot find filesystem info for device \"tank/podman/kind-probe/graph\""
kubelet.go "Failed to start ContainerManager"
err="failed to get rootfs info: cannot find filesystem info for device ..."
```

ContainerManager never starts, kubelet exits 1, the API server never comes up.

`master` (0e107a38) changes that error path to `return nil, fs.ErrFallbackToVFS`,
which routes to the VFS/statfs plugin and yields valid stats — but it is
unreleased.

## Verification the fix closes it

Checked out at cmosetick/cadvisor (verbatim master, HEAD 6a0c4f25):

- `lib/fs/zfs/plugin_test.go` `TestGetStatsFallsBackOnZfsError` passes — it
asserts exactly our node condition (`/dev/zfs` present + `getZfsStats` error
-> `ErrFallbackToVFS`).
- Added an end-to-end test driving `GetFsInfoForPath` with a ZFS partition whose
plugin returns `ErrFallbackToVFS`; master produces a valid `Fs` via statfs
fallback (non-zero Capacity). On v0.56.2/v0.60.5 the same input yields no `Fs`
-> the "cannot find filesystem info for device" crash.

Full evidence writeup (node output, v0.56.2-vs-master diff, repro) and the
end-to-end test source: https://gist.github.com/cmosetick/cd722e7f5897348046778432c19095db

## Environment

- released cadvisor affected: v0.56.2 (vendored by kubelet v1.36.1 /
kindest/node:v1.36.1); same code through v0.60.5.
- fix commit: 0e107a38 on master, in no tag.
- kind: v0.32.0, KIND_EXPERIMENTAL_PROVIDER=podman
- podman: 6.0.2 (rootful), storage driver = zfs
- host ZFS: zfs-2.2.2, kernel 6.8.0-136-generic

## Ask

Cut a tagged release including 0e107a38 so kubelet/kind can vendor a cadvisor
that no longer crashes on ZFS graph-driver nodes lacking the `zfs` CLI.

Related: #3904, #3918 (the fix), #3832/#3879 (separate ZFS double-count, still
present in stats.go).

Contributor guide

Open the contributing guide

Research direction

Review fs/zfs/plugin.go, commit 0e107a38, and lib/fs/zfs/plugin_test.go, then verify the fix is absent from tagged releases with git tag --contains 0e107a38. Follow the repository's release process to publish a tag containing the commit; done means downstreams can consume a release with the ZFS fallback and the cited test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, go, kubernetes, linux
Domain
devops, operating-systems, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.