rossoctl / rossoctl/serverless-harness
P4.1: implement microVM egress transport (vsock → shared proxy)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 7
- Avg merge
- 12h 59m
- Merged PRs (30d)
- 71
Description
Summary
Implement P4.1 — microVM egress transport — once its spec (PR #268) merges and the experiments that
gate it return. The spec is docs-only; nothing tracks the code. This issue is that code.
The spec is unusually explicit that this order is mandatory: "Build nothing before E12 answers."
(§9). So this issue exists to be blocked, and to hold the file list so a planner starting cold does
not re-derive it.
Why it is needed at all
The P4 microVM tier has no network device — no tap, no /network-interfaces call, no NAT anywhere
in deploy/microvm/. The only guest↔host channel is vsock, where the guest agent is parked in
accept() on port 1024 (build-snapshot.sh:561). Z5 assumes HTTPS_PROXY reaches the proxy over
ordinary IP networking because Z5's sandbox is a pod. On this tier Z5's mechanism is unreachable for
a purely mechanical reason, and curl / gh / MCP-over-HTTP cannot work today.
This is not a new egress design. Z5 settled forward proxy + baked CA + TLS interception +
placeholder-swap + allowlist-as-exfil-boundary in June (E1–E11) and RC1 implemented a static slice.
P4.1 changes only what sits behind 127.0.0.1:3128 and inherits every one of those decisions.
Blocked on
| Gate | State | If it fails |
|---|---|---|
| PR #268 merged (the spec) | open | — |
| #271 / E12 — does guest-initiated vsock on a second port survive snapshot restore? | open | T1 is dead. Fall back to per-clone netns + veth + double-MASQUERADE — 128 netns and 128 veth pairs at E11's ladder top, from a Firecracker doc that disclaims itself as "not a performant or secure setup" |
| #280 / E13 (run: PR #273) — root-cause E12's rung C@128 dropped connections | run, follow-up open | E13 falsified the memory hypothesis and T1 stands. Its open half is CPU/scheduling contention from neighbouring work — so design against that, not against a restore-count ceiling |
| #266 — nested/metal ratio | open | Supplies E13/E14 substrate; its question 3 (does nesting move the knee?) gates E14 on nested |
E12 is a boolean — no ladder, no baseline, no threshold — and runs on nested-m8i, not metal, for a
correctness reason: its guest needs a vsock-initiating client in the rootfs, which changes the rootfs
digest, and the authoritative E10/E11 numbers were legitimised by that digest being verified identical
(sha256:668af5893e9c70ef). Folding the helper into a metal snapshot would invalidate that comparison
invisibly — exit 0, no error, wrong result.
Scope (spec §12.1, settled — real paths)
- Loopback listener →
remote-worker/cmd/guest-agent, and not throughinternal/guestagent's
framed protocol: egress is opaque byte-splicing, not framed requests. - Socket creation, standby verification, teardown ordering →
remote-worker/internal/vmpool/.
The host must pre-create the socket, so it is created at replenishment, off the hot path, and a VM
becomes a standby only after its listener verifies. - CA public cert, the constant placeholder,
HTTPS_PROXY→ image-shipped constants in
build-snapshot.sh, so none of them violates P4 §5.2's no-secrets-and-nothing-unique-in-the-snapshot
invariant. - Guest loopback →
CID 2:1025→<jail>/run/v.sock_1025. - All Go. Nothing in
packages/is implied.
The owed decision to resolve first (spec §12.2)
What is the shared proxy, concretely? RC1's Profile B is a Kubernetes sidecar
(deploy/knative/sandbox-pool-ab2.yaml); this tier has no pods and no sidecars, and T3 makes the proxy
shared rather than co-located — so RC1's deployment shape does not transfer even though its plugin
semantics do.
The discriminating question, to answer before writing code: can AuthBridge accept a vsock listener and
a non-header identity source without being forked?
- Yes → run it as a host process.
- No → a thin shim in this repo isolates the incompatibility. This is the presumptive answer.
- A new proxy only wins if Z5's semantics turn out not to fit a shared non-Kubernetes proxy at all —
which would be a finding about Z5 and should be written up as one, not absorbed here.
Per-caller identity — and the Z1 boundary
A shared proxy forces per-caller identity, which is Z1's question. E11 measured Σ PSS 0.41 GB for
128 resident microVMs, so 128 sidecars would each cost more than the VM they serve — per-sandbox does
not survive density.
The jail socket path supplies identity: per-VM, and unforgeable from inside the guest — exactly the
property exchange.ts:41-46 records as missing. Ship it behind a CallerIdentity interface whose
first implementation is the socket path and whose second may be SPIFFE, so Z1 keeps the architectural
question rather than having it settled by a tier-local spec. §5 already states the cost plainly: an
abstraction with one implementation.
Acceptance
- The §12.2 proxy decision made and recorded before implementation starts
- Guest loopback listener in
cmd/guest-agent, outside the framed protocol - Socket pre-created at replenishment; standby admitted only after its listener verifies
- Teardown ordering handles a VM destroyed mid-connection without leaking sockets — note #255
already tracks two empty dirs leaking per VM id -
CallerIdentityinterface with the jail-socket-path implementation; no SPIFFE assumption baked in - CA cert / placeholder /
HTTPS_PROXYshipped as constants; P4 §5.2 invariant re-verified - Audit-log redaction rule implemented — the log records full URLs and some upstreams carry
secrets in query strings (§11) - Consequences from §11 carried into the code's docs, not silently dropped: host-wide egress blast
radius (T3), an interface with one implementation (T4), and no cert-pinning escape hatch —
Z5 §5.2 permits routing around the proxy on a documented exception; on this tier that exception
does not exist. A strict, unmitigated regression against Z5, and it must be recorded as one - No claim that the proxy cannot be escaped (matching P4 §1's posture on the VMM)
A bonus worth asserting in a test
With no network device, Z5 §5.1's concession that the model "can address any host… ignore any wrapper"
stops applying, and §8.6's reliance on NetworkPolicy is unnecessary. Z5 E4's "curl to anywhere is
structurally impossible" becomes literally true on this tier rather than policy-enforced (T7). That is
a testable property — assert it rather than asserting the policy.
Related: PR #268 (spec), #271 / PR #272 (E12), #273 (E13), #266 (substrate), #267 (session-scoped
working memory, out of scope for P4.1), #255 (VM-id dir leak), rossoctl/cortex#905 (Z5 injector).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
First wait for PR #268, E12, E13, and #266, then resolve the §12.2 proxy decision before coding. Read the spec and inspect remote-worker/cmd/guest-agent, remote-worker/internal/vmpool/, build-snapshot.sh, and deploy/knative/sandbox-pool-ab2.yaml. Done means the listed acceptance checks pass, including socket lifecycle, CallerIdentity, redaction, and the no-network-device assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, shell
- Domain
- infrastructure, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100