rossoctl / rossoctl/serverless-harness

microVM: two empty dirs leak per VM id, and a sudo-glob hazard when probing jail internals

Open
#255 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
7
Avg merge
12h 59m
Merged PRs (30d)
71

Description

Two follow-ups observed while clearing the bare-metal box after the authoritative E10/E11
runs (PR #251). Neither affected a measurement, and neither is a data or process leak — both
are hygiene/guidance items deliberately left out of that PR because they were outside its
scope.

1. Two empty directories leak per VM id, unbounded across runs

firecrackerVM.Destroy removes the jail contents correctly, but one level too deep:

// launcher_firecracker.go:218
jailRoot := filepath.Join(l.opts.ChrootBase, filepath.Base(l.opts.FirecrackerBin), req.ID, "root")
// launcher_firecracker.go:692 (in Destroy)
if err := os.RemoveAll(v.jailRoot); err != nil { … }

jailRoot is <ChrootBase>/firecracker/<id>/root, so Destroy reclaims everything inside
but leaves the per-VM parent <ChrootBase>/firecracker/<id> behind as an empty directory.

Separately, the per-VM cgroup directory is created by jailer (via --cgroup) and is removed
only by SweepOrphansremoveCgroupDir. Nothing on the normal destroy path rmdirs it,
so a clean shutdown also leaves one empty cgroup dir per VM id.

Measured on metal after two E10 runs (ITERS=200) plus one 14-rung E11 sweep:

leftover count content size
/srv/jail/firecracker/vm-N/ 3007 0 of 3007 held any content ~12 MB of inode
/sys/fs/cgroup/microvm.slice/microvm-vms.slice/vm-N/ 3007 0 procs in all 3007

So the reclamation that matters works: no VM data, no processes, nothing to attribute to a
run. What accumulates is ~4 KB of inode per VM id, and VM ids are minted monotonically, so it
grows without bound across runs on a long-lived host.

The METAL-RUNBOOK.md §3a preflight already clears both, so the operational path is covered
and no run is at risk. Suggested fix: have Destroy remove the per-VM parent dir rather than
<id>/root, and rmdir the per-VM cgroup on the normal destroy path (keeping SweepOrphans
as the orphan backstop, not the only reaper). Note the cgroup half must stay rmdir, never
rm -rf — see the D5 note in cgroup.go:359.

2. Probing jail internals: expand globs inside the privileged shell

A verification hazard rather than a defect, worth a caution in §3a because that section is
where preflight checks get extended.

$ sudo ls -1d /srv/jail/firecracker/vm-*     ->  0 directories
$ sudo find /srv/jail -maxdepth 3 -name 'vm-*' | wc -l   ->  3007

The shell expands vm-* before sudo runs. jailer creates
/srv/jail/firecracker as drwx------ root, so an unprivileged shell matches nothing and
hands sudo ls the literal unexpanded pattern — which then reports an absence. Adding sudo
to the front of the command does not fix it, which is why it survives review. I briefly drew
the wrong conclusion ("0 of 3007 jail dirs hold content") from a probe that had enumerated
nothing at all; the corrected probe gave the same answer for a real reason.

The shipped commands are correct and need no change — I checked. §3a's
sudo rm -rf "$SH_WORKSPACE_ROOT"/* /srv/jail/firecracker works because /srv/workspaces is
drwxr-xr-x (verified empirically on the box: the glob expands, the decoy dir is removed),
and the jail path carries no wildcard. The hazard is confined to paths jailer creates at
mode 0700 — i.e. anything under /srv/jail/firecracker/<id>/.

Suggested fix: one caution in §3a — when probing jail internals, glob inside the privileged
shell (sudo bash -c '…') or let a privileged walker expand it (sudo find), and treat an
absence reported by an unprivileged glob as unproven. This is the same family as the existing
[ -S "$sock" ]-reads-absent and kill -0-reads-dead artifacts already documented on the
branch.


Context: both were found on 2026-09-15 while returning srv-r16b14s16 to a clean state after
the metal run. The box is left provisioned (toolchain + verified golden snapshot) so a rerun
is git fetch + reset rather than a rebuild.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with launcher_firecracker.go at the jailRoot construction and Destroy, then read cgroup.go:359 and METAL-RUNBOOK.md §3a. Check the normal destroy and orphan-cleanup paths, and review the privileged glob guidance. Done means per-VM jail and cgroup directories are reclaimed safely while the runbook warns that unprivileged glob expansion cannot prove absence.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, shell
Domain
devops, documentation, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.