feat: support in-place supervisor upgrade via re-exec

Problem Statement

Upgrading the openshell-sandbox supervisor today requires destroying and recreating the sandbox, which kills the agent workload running inside it. We want the supervisor to be able to replace its own binary in place — bring in a new executable, execve() it, and continue supervising the same child processes — analogous to systemctl daemon-reexec. Because execve destroys the entire address space, this requires serializing supervisor state so the incoming binary can resume rather than bootstrap.

This spike assesses feasibility, maps the exact state that must cross the exec boundary, and identifies the design decisions that need human input before implementation.

Technical Context

openshell-sandbox runs inside the workload container in one of three topologies: combined (--mode=network,process, usually PID 1, root with SYS_ADMIN/NET_ADMIN/SYS_PTRACE/SYSLOG), K8s sidecar (two supervisors — non-root --mode=network plus --mode=process — coupled by a one-shot Unix control socket), and --mode=network-init (one-shot, exits immediately).

The structural fact that dominates this design: privileged bootstrap is strictly ordered before a one-way seccomp door. Everything requiring mount/bpf/pivot_root/kexec_* happens during startup, and then apply_supervisor_startup_hardening() (crates/openshell-supervisor-process/src/run.rs:111) installs a filter that is inherited across both fork and execve. A re-exec'd supervisor therefore comes up already hardened and cannot redo those bootstrap steps.

Feasibility verdict: yes, with a hard boundary. execve preserves what the kernel owns and destroys what userspace holds. That split determines the entire design.

Affected Components

Technical Investigation

Architecture Overview

Control flow: main() (main.rs:457) → tracing setup → multi-thread tokio runtime (main.rs:513) → run_sandbox() (lib.rs:89) → run_networking() (supervisor-network/src/run.rs:76) and run_process() (supervisor-process/src/run.rs:53). run_sandbox blocks on the entrypoint child's exit and returns its code, which becomes the container exit code (main.rs:643).

What survives execve for free: PID identity — so the entrypoint and SSH-session processes remain direct children (single fork+exec, confirmed at process.rs:730 and ssh.rs:870); netns membership; cgroup; nftables rules; mount-namespace membership; no_new_privs; and any fd with FD_CLOEXEC cleared.

Also load-bearing: execve runs no Rust destructors, so kill_on_drop(true) (process.rs:603), ProxyHandle::drop (proxy.rs:402), and NetworkNamespace::drop (netns/mod.rs:359) never fire. That is exactly what we want here — and also why a failed resume leaks the netns.

What dies: all tokio state and all TLS/SSH crypto state. Established SSH sessions (russh state is userspace) and in-flight proxy connections (CONNECT tunnels, TLS-terminated L7 relays) cannot be handed to a new process image under any design short of moving them into separate processes.

Capability caveat. execve recomputes the permitted capability set. For a euid-0 process the kernel's legacy-root rule restores it, so the combined topology is fine. But the K8s network sidecar runs as a non-root uid with SYS_PTRACE + DAC_READ_SEARCH (driver-kubernetes/src/driver.rs:1743-1766), and there are no ambient capabilities configured anywhere in this repo. On re-exec those capabilities would be lost, and they are needed at runtime by procfs::read_process_binary (supervisor-network/src/procfs.rs:93-160), which fails hard when the exe symlink is unreadable — the sidecar would silently lose binary-aware policy enforcement and start failing closed. This claim is kernel- and runtime-config dependent and should be verified empirically before designing around it.

Code References

Current Behavior

1. main() dispatches copy-self / debug-rpc pre-clap, parses Args, handles --mode=network-init, builds the rolling appender and tokio runtime, calls run_sandbox, then std::process::exit.
2. run_sandbox(): set_ctx OCSF context → (process-only sidecar) connect_process_client, fatal on failure → load_policy (5 retries then fatal), build OpaEngine, install middleware registries → resolve_process_identity → provider env fetch → create_netns_for_proxy (ip netns add, veth, routes, nft bypass rules via nsenter) → run_networking (symlink task, BinaryIdentityCache, SandboxCa::generate(), ProxyHandle::start_with_bind_addr binding 10.200.0.1:3128) → (network sidecar) sidecar_control::spawn_server → aggregators + policy poll loop → optional GCE metadata listener → run_process.
3. run_process(): passwd/group updates, identity validation, prepare_filesystem_with_identity, install_initial_agent_skill, apply_supervisor_startup_hardening(), bypass monitor, PID-limit check, zombie reaper, SSH server + readiness gate, supervisor_session::spawn, ProcessHandle::spawn, wait_for_process_exit_or_shutdown.
4. Exit: entrypoint status → drop(networking) → main exits. NetworkNamespace::drop tears down the netns.

What Would Need to Change

Resume disposition matrix — the core deliverable:

Per-component work:

• openshell-sandbox/main.rs: a pre-clap resume --state-fd <n> and verify-state subcommand alongside copy-self. A state fd is preferable to a state path — no writable file, no TOCTOU, no path the workload can point at.
• openshell-sandbox/lib.rs: run_sandbox is a ~700-line straight-line function with interleaved conditionals. It must be split into bootstrap() and resume() sharing a runtime-assembly step, or every SKIP/REDO decision becomes an if resuming branch inside an already-dense function. This refactor is the single largest cost and should land first, independently.
• New crates/openshell-sandbox/src/handoff.rs: a versioned HandoffState + fd table, serialized over a memfd/pipe, with an explicit schema version and an unknown-field rejection policy.
• supervisor-process/run.rs: an AdoptedEntrypoint variant that skips both ProcessHandle::spawn and the hardening prelude.
• supervisor-process/process.rs: ProcessHandle needs a constructor backed by pidfd_open(2) (available post-prelude) or a waitpid loop; Drop (:899) must not unregister an adopted child during handoff.
• supervisor-process/managed_children.rs: add hydrate(&[i32]).
• supervisor-process/ssh.rs: ssh_server_init (:44) accepts a pre-bound UnixListener and a persisted host key.
• supervisor-process/netns/mod.rs: NetworkNamespace::adopt(name) that re-opens /var/run/netns/<name> without ip netns add, plus suppression of Drop teardown during handoff.
• supervisor-network/run.rs: SandboxCa must become adoptable — the biggest correctness change on the network side.
• supervisor-network/proxy.rs: accept a pre-bound TcpListener.
• supervisor-network/identity.rs: BinaryIdentityCache export/import.
• supervisor-network/opa.rs: carry the generation counter forward so numbering stays monotonic.
• Drivers: a writable, workload-inaccessible location for the replacement binary plus digest verification. No such location exists today in any driver.

Alternative Approaches Considered

(a) Full state handoff with fd passing. Serialize HandoffState + fd table; execve with FD_CLOEXEC cleared. Preserves the proxy listener backlog, SSH listener, sidecar control channel, netns fd, and (if the CA key is carried) TLS trust continuity. In-flight TLS/SSH crypto state still dies regardless. Cost: highest — ~25-40 files, requires the run_sandbox refactor and a versioned wire format, and moves the CA private key across a serialization boundary.

(b) Minimal re-exec: keep the entrypoint, drop everything session-shaped. Serialize only what is unrecoverable or dangerous to re-derive: entrypoint PID + managed children, policy proto + generation + last-known-good flag, provider credential snapshot, netns name, CA key + cert, TOFU map. Rebind the proxy and SSH listeners; let the gateway session reconnect on its own. Accept dropped SSH sessions, dropped proxy connections, a changed SSH host key, and seconds of degraded egress. The right first increment for the combined topology. Explicitly excludes the K8s sidecar topology.

(c) Split supervisor: thin supervising parent + replaceable worker child. A tiny never-upgraded PID-1 parent owning the netns, listeners, and entrypoint; a worker child holding the replaceable logic. Pros: eliminates the PID-1 fatality risk structurally (the parent can retry or roll back), the worker can be Landlock-restricted, and state handoff becomes an ordinary fd/socket protocol instead of an exec-time trick. Cons: the entrypoint becomes a grandchild of PID 1 unless the parent spawns it, changing reaping (run.rs:146-204), the expected_ssh_peer_pid check (supervisor_session.rs:699), and every /proc/<pid> identity assumption in procfs.rs. Larger change, simpler steady state. Worth deciding on before committing to (a).

Relationship to RFC #981 ("Split Supervisor and Agent into Separate Pods with gVisor Isolation"). That RFC proposes a pod-level split for isolation reasons; option (c) here is a process-level split within the supervisor for upgradability. They are not the same proposal, but they push in the same architectural direction and would interact: the split-pod model already sideloads a trusted OpenShell binary into the agent pod as its entrypoint, which is a second binary-delivery path this feature would need to account for. If #981 is accepted, option (c) becomes substantially more attractive and this spike should be re-scoped against it rather than built independently.

(d) Drain and recreate the sandbox. Zero new supervisor code and no writable-binary surface, but it kills the workload — precisely what this issue exists to avoid.

Patterns to Follow

• Pre-clap subcommand dispatch matching copy-self / debug-rpc (main.rs:461-481).
• OwnedFd where ownership transfers, RawFd only where existing code already does; clear FD_CLOEXEC with the existing fcntl idiom (bypass_monitor/mod.rs:562-568).
• Readiness signalling via oneshot, as the SSH server does (run.rs:239, ssh.rs:134), so resume failures surface as errors rather than hangs.
• OCSF per AGENTS.md: ConfigStateChangeBuilder for re-exec initiated/completed/state-adopted, AppLifecycleBuilder for the new supervisor coming up, dual-emitted DetectionFindingBuilder for binary-verification failure. Never log the CA key or credential material.
• Graceful-degradation logging (lib.rs:244-262) for non-fatal adoption failures — and its inverse: anything that would silently loosen policy must be fatal, not degraded.
• Fail-closed on missing prior state, mirroring lib.rs:2020-2024.
• #[allow(unsafe_code)] with an explicit SAFETY comment on every libc call.

Proposed Approach

Sequence this rather than building the full handoff up front. First, land the run_sandbox split into bootstrap/assembly phases as an independent refactor — it is separately valuable and separately testable, and every later step depends on it. Second, persist the SSH host key, which is correct regardless of whether re-exec ships. Third, introduce a versioned HandoffState plus a verify-state subcommand with no exec wired up, so the serialization format can be validated in isolation. Fourth, implement combined-topology re-exec along option (b), adopting the entrypoint PID, netns, CA, and TOFU map while letting sessions drop. Sidecar-topology coordination comes last and is gated on a maintainer decision about the one-shot control socket.

Scope Assessment

• Complexity: High — not because any single piece is hard, but because correctness depends on getting a ~30-item SKIP/REDO/ADOPT matrix right across three topologies, and the largest prerequisite is refactoring a 700-line function that currently encodes startup order implicitly.
• Confidence: High for the code-behavior claims (each is line-cited). Medium for the execve capability-transformation claim and for the blast radius of CA regeneration on real workloads.
• Estimated files to change: 12-18 for combined-topology-only (option b); 25-40 including sidecar, drivers, Helm, proto, gateway, and docs.
• Issue type: feat, preceded by a refactor PR splitting run_sandbox.

Risks & Open Questions

1. PID-1 fatality with no rollback. Every fatal path in the current startup sequence becomes a workload-killing path on resume — load_policy retry exhaustion (lib.rs:1906), SSH bind failure (run.rs:283-295), sidecar control checks (lib.rs:713-753). Needs an explicit design answer. Mitigations: a verify-state dry run in a forked child before the point of no return; the outgoing supervisor validates the new binary's version and schema acceptance before exec; option (c) removes the risk structurally.
2. Ephemeral CA regeneration breaks the running workload's TLS. SandboxCa::generate() runs on every start (supervisor-network/run.rs:207-272). Long-lived processes (Node, Python requests, Java) read the CA bundle at startup via SSL_CERT_FILE/NODE_EXTRA_CA_CERTS; after a resume with a fresh CA, every TLS request they make through the proxy fails validation. Either serialize the CA private key — a supervisor-private key crossing a serialization boundary — or accept the breakage.
3. Binary delivery is the whole security story. Today no writable executable path exists in any driver (K8s readOnly: true, Docker :ro,z, Podman rw: false), and nothing authenticates the binary beyond image-pull integrity. Introducing a writable path is CWE-494 territory. Requires: writable only by a non-workload identity, digest verification against a gateway-minted allow-list, and an OCSF DetectionFinding on mismatch. If the trigger is gateway-initiated and LLM-influenceable, assess prompt-injection reachability (OWASP LLM06). Note the marginal risk of the exec primitive itself is near zero — memfd_create and execveat are already unblocked in the prelude.
4. Capability loss on non-root re-exec. If confirmed, the K8s network sidecar cannot use this mechanism without ambient capabilities, which is itself a hardening regression. Verify empirically.
5. Sidecar one-shot control socket makes independent re-exec impossible. Changing it weakens a deliberate anti-impersonation property.
6. Entrypoint exit during the handoff window. The child can exit between the outgoing supervisor's last wait and the incoming supervisor's first waitpid. The exit status must not be lost or stolen by the new reaper — managed_children hydration must happen before the reaper starts.
7. TOFU cache loss re-trusts binaries. A binary swapped during the handoff window would be re-trusted at its new hash. Narrow but real integrity weakening.
8. State blob as an untrusted input. Even fd-passed, the deserializer runs as root pre-Landlock. Version skew is the point of the feature, so the format must reject unknown/mismatched schemas rather than best-effort parse.
9. Multi-threaded execve semantics. PID is retained even when a non-leader thread execs, but a thread stuck in an uninterruptible syscall can stall it. Drive the exec from a controlled point after quiescing.
10. Log continuity. The non_blocking appender's buffer is lost (main.rs:498-508). Acceptable, but operators debugging an upgrade will see a gap — document it.

Pre-existing defect encountered during this spike — already tracked

While mapping the mount-namespace machinery, this spike independently rediscovered that prepare_supervisor_identity_mount_namespace_from_env (process.rs:316) has zero callers repo-wide, so supervisor_identity_mount_from_env (:331-341) hard-fails entrypoint and SSH spawn whenever PROVIDER_SPIFFE_WORKLOAD_API_SOCKET is set in privileged mode. This is already filed as #2571, which has a live-cluster reproduction. This spike adds one datum to that report: the regression was introduced by 1ca23bc5 (#1650, the process/network subcrate split), which dropped the call site that previously sat at crates/openshell-sandbox/src/lib.rs:615 immediately before apply_supervisor_startup_hardening().

It is noted here only because it is adjacent code — it is not a blocker for this work, and fixing it is #2571's scope, not this issue's.

Disposition Readiness

• Assessment: The spike establishes that the feature is technically feasible, identifies the exact kernel/userspace boundary that constrains the design, produces a complete SKIP/REDO/ADOPT matrix with line-level evidence, and surfaces the decisions a human must make. That is sufficient for an accept/decline decision.
• Missing evidence: One empirical check would firm up the scope — whether a non-root execve actually loses SYS_PTRACE/DAC_READ_SEARCH in the K8s network sidecar (Risk 4). It affects whether the sidecar topology is in scope at all, but does not block disposition of the combined-topology work.
• Label note: the create-spike skill specifies state:validated / state:accepted / state:needs-info and agent:* labels, but none of those exist in this repository. Filed with state:review-ready, matching the convention used by other spike issues (#2157, #2144). The skill's label taxonomy appears to have drifted from the repo and is worth reconciling via sync-agent-infra.

Test Considerations

Existing infrastructure. Unit tests are colocated (#[cfg(test)] mod tests). Behavioral seccomp tests fork children and assert errnos (seccomp.rs:579-610). One integration test exercises the real binary (crates/openshell-sandbox/tests/stdout_logging.rs, via CARGO_BIN_EXE_openshell-sandbox). E2E lives in e2e/rust/tests/ (mise run e2e) with directly relevant precedents: gateway_resume.rs, podman_gateway_resume.rs, vm_gateway_resume.rs, local_driver_token_restart.rs, live_policy_update.rs, bypass_detection.rs.

Levels needed:

• Unit: HandoffState round-trip; version-skew rejection (older→newer, newer→older, unknown fields); managed_children::hydrate; NetworkNamespace::adopt; the SKIP/REDO/ADOPT decision table as a pure function so it is testable without a container.
• Integration (real binary): verify-state accepts a valid blob and rejects a malformed or foreign-version one; resume with no state fails closed.
• E2E (new, modelled on gateway_resume.rs): start a sandbox with a long-running workload process, trigger re-exec, then assert (i) the entrypoint PID is unchanged, (ii) egress still works and is still policy-enforced, (iii) a denied destination is still denied, (iv) the entrypoint exit code is correctly reported after re-exec, (v) the netns and veth were not recreated. Plus a negative test: re-exec into a deliberately broken binary must not kill the workload — this is the test that validates the rollback design.
• Seccomp: a behavioral test proving the prelude is still enforced after re-exec and that mount(2) still returns EPERM in the resumed process. This is the regression that would silently reopen the one-way door.

Infrastructure gaps. There is no harness for "run the supervisor, replace its binary, re-exec it" — the existing e2e resume tests restart the gateway, not the supervisor. Something new is needed, likely a test-only subcommand or env-gated trigger plus a second build artifact to exec into.

LSM fragility. Any new test that forks and execs a system binary and then reads /proc/<pid>/exe will fail on SELinux-enforcing hosts (domain transition makes the link ENOENT, not EACCES). Follow the existing guard at bypass_monitor/mod.rs:592-602. For re-exec tests, prefer exec'ing the same binary — same label, no transition — which is also the real-world case.

LSM impact on the feature itself. A replacement binary written into a new writable volume inherits that volume's label; on container-selinux hosts an emptyDir is container_file_t and executable by container_t, but a host bind-mount without :z/:Z may not be. The K8s driver sets per-container appArmorProfile (driver.rs:1792,1848,2551), which can deny x on the new path — document the required rule. /proc/<pid>/exe across domains is already load-bearing in procfs.rs:93-160; the resumed supervisor must treat ENOENT as "unreadable", not "gone".

Documentation impact. architecture/sandbox.md "Startup Flow" (lines 24-35) and "Runtime Model" describe a single-shot startup and would need a resume path. docs/reference/gateway-config.mdx needs updating only if the feature introduces gateway TOML keys (e.g. an allowed-supervisor-digest list); driver-level changes would land near gateway-config.mdx:324-329. Per AGENTS.md's Cluster Infrastructure rule, a writable supervisor volume in the pod spec makes updating debug-openshell-cluster mandatory; helm-dev-environment and openshell-cli need updating if the pod spec or a trigger command changes.

Created by spike investigation. state:validated means the issue is ready for human disposition. A human applies state:accepted if OpenShell should pursue the work and places it on the roadmap separately. To queue unattended agent planning, a human applies agent:plan-requested; a direct request to an agent does not require that label.

Start with crates/openshell-sandbox/src/main.rs:457 and lib.rs:89, then trace the process and network startup paths listed in the affected components. Map which state can be skipped, redone, or adopted across execve, verify the non-root capability caveat empirically, and document the feasibility boundary and design decisions needed before implementation.