Verified: 0e107a38 (fs/zfs VFS fallback) fixes a fatal kubelet crash on the ZFS graph driver, but is unreleased — request a release
- Dominant language
- Go
- Stars
- 19.4k
- Forks
- 2.5k
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 1
Description
## Summary
Commit 0e107a38 ("fs/zfs: fall back to VFS when zfs stats are unavailable",
part of the #3918 fix) is on `master` but is **not in any tagged release**
(`git tag --contains 0e107a38` is empty; it landed after v0.60.5). Because
kubelet vendors a released cadvisor, downstreams still hit a fatal kubelet
startup crash on ZFS-backed nodes. Requesting a release be cut so the fix is
consumable.
## Why it matters (the failure the fix closes)
On a Kubernetes node whose container store uses the **podman/containerd ZFS
graph driver** (e.g. a `kind` node run under podman with
`storage.driver = "zfs"`), `/dev/zfs` is present in the node but the `zfs` CLI
is **not installed**. cadvisor's `GetZfsStats` shells out to `zfs list ...`,
which fails with exec ENOENT (exit 127).
In released cadvisor (v0.56.2, as vendored by kubelet v1.36.1 — and unchanged
through v0.60.5), `fs/zfs/plugin.go GetStats` does:
```go
capacity, free, avail, err := getZfsStats(device)
if err != nil {
return nil, err // <-- no VFS fallback
}
```
So no `Fs` is produced for the ZFS rootfs partition, and the kubelet dies:
```
container_manager_linux.go "Unable to get rootfs data from cAdvisor interface"
err="cannot find filesystem info for device \"tank/podman/kind-probe/graph\""
kubelet.go "Failed to start ContainerManager"
err="failed to get rootfs info: cannot find filesystem info for device ..."
```
ContainerManager never starts, kubelet exits 1, the API server never comes up.
`master` (0e107a38) changes that error path to `return nil, fs.ErrFallbackToVFS`,
which routes to the VFS/statfs plugin and yields valid stats — but it is
unreleased.
## Verification the fix closes it
Checked out at cmosetick/cadvisor (verbatim master, HEAD 6a0c4f25):
- `lib/fs/zfs/plugin_test.go` `TestGetStatsFallsBackOnZfsError` passes — it
asserts exactly our node condition (`/dev/zfs` present + `getZfsStats` error
-> `ErrFallbackToVFS`).
- Added an end-to-end test driving `GetFsInfoForPath` with a ZFS partition whose
plugin returns `ErrFallbackToVFS`; master produces a valid `Fs` via statfs
fallback (non-zero Capacity). On v0.56.2/v0.60.5 the same input yields no `Fs`
-> the "cannot find filesystem info for device" crash.
Full evidence writeup (node output, v0.56.2-vs-master diff, repro) and the
end-to-end test source: https://gist.github.com/cmosetick/cd722e7f5897348046778432c19095db
## Environment
- released cadvisor affected: v0.56.2 (vendored by kubelet v1.36.1 /
kindest/node:v1.36.1); same code through v0.60.5.
- fix commit: 0e107a38 on master, in no tag.
- kind: v0.32.0, KIND_EXPERIMENTAL_PROVIDER=podman
- podman: 6.0.2 (rootful), storage driver = zfs
- host ZFS: zfs-2.2.2, kernel 6.8.0-136-generic
## Ask
Cut a tagged release including 0e107a38 so kubelet/kind can vendor a cadvisor
that no longer crashes on ZFS graph-driver nodes lacking the `zfs` CLI.
Related: #3904, #3918 (the fix), #3832/#3879 (separate ZFS double-count, still
present in stats.go).
Contributor guide
Research direction
Review fs/zfs/plugin.go, commit 0e107a38, and lib/fs/zfs/plugin_test.go, then verify the fix is absent from tagged releases with git tag --contains 0e107a38. Follow the repository's release process to publish a tag containing the commit; done means downstreams can consume a release with the ZFS fallback and the cited test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, go, kubernetes, linux
- Domain
- devops, operating-systems, release
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 32/100