rossoctl / rossoctl/serverless-harness
E9: run the deployment-tier comparison — VM supervisor+mux vs Knative one-session-per-pod
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 7
- Avg merge
- 12h 59m
- Merged PRs (30d)
- 71
Description
Summary
Run E9 — deployment-tier comparison (P6 §5.3): the VM supervisor+mux arm against the Knative
one-session-per-pod arm, same host class and same workload, to isolate what the deployment tier
costs. No issue tracks this today; the epic lists it as outstanding and it is the only P6 experiment
with neither a spec-amendment issue nor a driver issue.
E9 compares capacity, not cost. E1-style economics on the VM is round two (§7).
The two pins — and the second one is easy to get wrong
Isolating one tier means pinning every other. E9 has two such constraints:
1. The model tier. E6's existing numbers were taken against a real model and are not comparable
to stub-driven ones. The Knative arm must be re-run against the same stub — cheap, via
ANTHROPIC_BASE_URL on the ksvc, and E6's driver already manipulates and restores ksvc env
(restore_ksvc_env on its EXIT trap).
2. The tool tier. Left to their defaults the arms do not match: the Knative arm resolves pods
and gets persistentExecInPod's fast channel, while the VM arm runs over gRPC and has none (§3.1a).
The comparison would then charge the deployment tier for a transport difference, biased against the
VM. So both arms run the relay + gRPC transport — SH_REMOTE_SANDBOX=1 against the in-cluster
relay, which relay-leaf-smoke.sh already exercises. Configuration, not code.
The invariant a reviewer should check: E9 varies the deployment tier and nothing else. Any future arm
inherits both pins.
Preconditions
| # | Gate | Why |
|---|---|---|
| 1 | E8 run first | E9 reuses E8's rungs, stub and knee-detection; comparing two tiers before either arm's own knee is understood produces a ratio nobody can attribute |
| 2 | #254 merged | Same two-arm methodology and the same reporting discipline apply here |
| 3 | PR #253 merged | Both arms need /turn tool calls actually reaching the sandbox pool |
| 4 | Model stub deployable to the cluster | It lives at deploy/knative/model-stub/ — not under deploy/vm/ — precisely because both arms must drive the same stub (§5.4). The VM arm runs the same image locally |
| 5 | Same duty basis row on both arms | §2.3, named in each run record |
Step 7 of P6 §3's build order is exactly this: "E9 second arm on a cluster — both arms against the
same stub and the same gRPC transport."
Acceptance
- Both arms run on the same host class, same workload, same duty-basis row
- Both arms recorded as running the relay + gRPC transport; neither arm on
persistentExecInPod - Both arms against the same stub, with the profile recorded and pinned per arm
- Connections per session recorded on every rung of both arms and required to match across
arms (§3.4) — a mismatch charges one policy for the other's connection budget - The reported figure is a deployment-tier comparison, with the model and tool tiers shown pinned
- Per-arm knee attribution, not just a ratio
- No cost/resource-seconds claim (out of scope per §7)
- Results in
deploy/vm/EXPERIMENTS.md, pointer fromdocs/experiment-results.md
Note on P5
Like E8, E9 is measurable single-subject (§3.6). Mixed-subject rungs and the isolation claim need
#239; the comparison itself does not. State which was run.
Related: E8 run, #254, PR #253, #240 (ScaledJob → elastic pod pool, which changes what the Knative
arm is — if #240 lands first, record which deployment model the cluster arm measured).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by checking E8, #254, and PR #253, then inspect deploy/knative/model-stub/, the E6 driver’s restore_ksvc_env handling, and relay-leaf-smoke.sh. Run both deployment arms only after the listed preconditions, recording matching transport, stub, workload, duty basis, connections per session, and per-arm knees. Put results in deploy/vm/EXPERIMENTS.md and link them from docs/experiment-results.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100