P4: worker-process parallelism — does N microvm-worker processes give N× throughput?

Summary

E11's density sweep (c = 1…64, knee at c=8, bound: replenishment) ran its entire ladder against exactly one microvm-worker process — confirmed directly from deploy/microvm/e11-density.sh (start_microvm_stack starts one worker per (D, GuestRAMBytes) slice and the whole ACTIVE_RUNS sweep runs against it before teardown). This issue asks whether running multiple independent worker processes on the same host multiplies the ~39-43 execs/sec ceiling, or whether some shared host-level resource (snapshot I/O, the pinned memfile, virtiofsd, cgroup hierarchy) caps the host, not the process.

Why this matters

• This is the item most directly tied to the "multiple users, multiple sessions... optimized for throughput" goal at a service level. One worker's ceiling only matters for capacity planning if it's also the service's ceiling.
• Host resource headroom is enormous at the tested range (hostCpuFraction ~0.001 flat, Σ PSS ≤ 0.41 GB at 128 resident microVMs — see remote-worker/internal/vmpool/pin_linux.go's CoW-sharing comment), so there's no obvious resource reason a second worker couldn't run alongside the first. Whether it actually scales throughput is untested.
• The P6 density work already hit an analogous trap in a different tier: doubling worker count there moved the ceiling by -6%, because the host was already saturated and the bottleneck wasn't per-worker. Worth checking whether the same trap applies here before assuming linear scaling.

What's already known / where to look

• Driver: deploy/microvm/e11-density.sh, specifically start_microvm_stack/stop_microvm_stack (one process per config slice, each with its own SANDBOX_ID, binding its own relay port) and the outer loop in main() that currently never starts two microVM stacks concurrently.
• Config: each worker process holds one vmpool.Config (remote-worker/internal/vmpool/config.go) — one SnapshotDir, one StandbyDepth, one GuestRAMBytes, fixed at process start. Two workers pointed at the same SnapshotDir would each independently PinMemoryFile it — worth checking whether that's safe/redundant (two MAP_SHARED+mlock mappings of the same file should just double-count the lock accounting, not the physical pages, but this hasn't been verified under concurrent worker processes).
• Host-level shared resources to watch for contention: the golden snapshot's memfile (page-cache reads), /srv/jail and /srv/workspaces (per-VM directories — see the "chroot base / run dir must share a device with the snapshot" fix round in deploy/microvm/EXPERIMENTS.md), the jailer/cgroup hierarchy (microvm.slice), and vsock connection setup if it goes through any shared kernel structure.

Open questions for the spec

1. Should the two (or N) workers share one SnapshotDir, or does each need its own copy to rule out I/O contention as a confound? (Sharing is more realistic for a production deployment; separate copies isolate the variable better.)
2. What's the right topology to test first — 2 workers at moderate concurrency each, or N workers each pinned to c=8 (the known-healthy single-worker operating point)?
3. Does this need its own relay/Redis instance per worker, or can they share the existing single relay from e11-density.sh?

Suggested scope for a first cut

Start two microvm-worker processes (same snapshot, same D/RAM config) on the same metal host, repeat the c-sweep against each independently and simultaneously, and compare combined throughput/latency against 2x the single-worker c=8 numbers (39.04/s → expect ~78/s if it scales cleanly).

Part of #256 (P4 epic: next experiments after the metal STOP).

Start with deploy/microvm/e11-density.sh, especially start_microvm_stack, stop_microvm_stack, and main(), then read remote-worker/internal/vmpool/config.go and the relevant device-sharing notes in deploy/microvm/EXPERIMENTS.md. Run two workers with the same snapshot and D/RAM configuration, repeat the c-sweep concurrently, and compare combined throughput and latency with twice the single-worker c=8 result while watching the listed shared host resources.