rossoctl / rossoctl/serverless-harness
microVM: two empty dirs leak per VM id, and a sudo-glob hazard when probing jail internals
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 7
- Avg merge
- 12h 59m
- Merged PRs (30d)
- 71
Description
Two follow-ups observed while clearing the bare-metal box after the authoritative E10/E11
runs (PR #251). Neither affected a measurement, and neither is a data or process leak — both
are hygiene/guidance items deliberately left out of that PR because they were outside its
scope.
1. Two empty directories leak per VM id, unbounded across runs
firecrackerVM.Destroy removes the jail contents correctly, but one level too deep:
// launcher_firecracker.go:218
jailRoot := filepath.Join(l.opts.ChrootBase, filepath.Base(l.opts.FirecrackerBin), req.ID, "root")
// launcher_firecracker.go:692 (in Destroy)
if err := os.RemoveAll(v.jailRoot); err != nil { … }
jailRoot is <ChrootBase>/firecracker/<id>/root, so Destroy reclaims everything inside
but leaves the per-VM parent <ChrootBase>/firecracker/<id> behind as an empty directory.
Separately, the per-VM cgroup directory is created by jailer (via --cgroup) and is removed
only by SweepOrphans → removeCgroupDir. Nothing on the normal destroy path rmdirs it,
so a clean shutdown also leaves one empty cgroup dir per VM id.
Measured on metal after two E10 runs (ITERS=200) plus one 14-rung E11 sweep:
| leftover | count | content | size |
|---|---|---|---|
/srv/jail/firecracker/vm-N/ |
3007 | 0 of 3007 held any content | ~12 MB of inode |
/sys/fs/cgroup/microvm.slice/microvm-vms.slice/vm-N/ |
3007 | 0 procs in all 3007 | — |
So the reclamation that matters works: no VM data, no processes, nothing to attribute to a
run. What accumulates is ~4 KB of inode per VM id, and VM ids are minted monotonically, so it
grows without bound across runs on a long-lived host.
The METAL-RUNBOOK.md §3a preflight already clears both, so the operational path is covered
and no run is at risk. Suggested fix: have Destroy remove the per-VM parent dir rather than
<id>/root, and rmdir the per-VM cgroup on the normal destroy path (keeping SweepOrphans
as the orphan backstop, not the only reaper). Note the cgroup half must stay rmdir, never
rm -rf — see the D5 note in cgroup.go:359.
2. Probing jail internals: expand globs inside the privileged shell
A verification hazard rather than a defect, worth a caution in §3a because that section is
where preflight checks get extended.
$ sudo ls -1d /srv/jail/firecracker/vm-* -> 0 directories
$ sudo find /srv/jail -maxdepth 3 -name 'vm-*' | wc -l -> 3007
The shell expands vm-* before sudo runs. jailer creates
/srv/jail/firecracker as drwx------ root, so an unprivileged shell matches nothing and
hands sudo ls the literal unexpanded pattern — which then reports an absence. Adding sudo
to the front of the command does not fix it, which is why it survives review. I briefly drew
the wrong conclusion ("0 of 3007 jail dirs hold content") from a probe that had enumerated
nothing at all; the corrected probe gave the same answer for a real reason.
The shipped commands are correct and need no change — I checked. §3a's
sudo rm -rf "$SH_WORKSPACE_ROOT"/* /srv/jail/firecracker works because /srv/workspaces is
drwxr-xr-x (verified empirically on the box: the glob expands, the decoy dir is removed),
and the jail path carries no wildcard. The hazard is confined to paths jailer creates at
mode 0700 — i.e. anything under /srv/jail/firecracker/<id>/.
Suggested fix: one caution in §3a — when probing jail internals, glob inside the privileged
shell (sudo bash -c '…') or let a privileged walker expand it (sudo find), and treat an
absence reported by an unprivileged glob as unproven. This is the same family as the existing
[ -S "$sock" ]-reads-absent and kill -0-reads-dead artifacts already documented on the
branch.
Context: both were found on 2026-09-15 while returning srv-r16b14s16 to a clean state after
the metal run. The box is left provisioned (toolchain + verified golden snapshot) so a rerun
is git fetch + reset rather than a rebuild.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with launcher_firecracker.go at the jailRoot construction and Destroy, then read cgroup.go:359 and METAL-RUNBOOK.md §3a. Check the normal destroy and orphan-cleanup paths, and review the privileged glob guidance. Done means per-VM jail and cgroup directories are reclaimed safely while the runbook warns that unprivileged glob expansion cannot prove absence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, shell
- Domain
- devops, documentation, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100