rossoctl / rossoctl/serverless-harness

E9: run the deployment-tier comparison — VM supervisor+mux vs Knative one-session-per-pod

Open
#276 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
7
Avg merge
12h 59m
Merged PRs (30d)
71

Description

Summary

Run E9 — deployment-tier comparison (P6 §5.3): the VM supervisor+mux arm against the Knative
one-session-per-pod arm, same host class and same workload, to isolate what the deployment tier
costs. No issue tracks this today; the epic lists it as outstanding and it is the only P6 experiment
with neither a spec-amendment issue nor a driver issue.

E9 compares capacity, not cost. E1-style economics on the VM is round two (§7).

The two pins — and the second one is easy to get wrong

Isolating one tier means pinning every other. E9 has two such constraints:

1. The model tier. E6's existing numbers were taken against a real model and are not comparable
to stub-driven ones
. The Knative arm must be re-run against the same stub — cheap, via
ANTHROPIC_BASE_URL on the ksvc, and E6's driver already manipulates and restores ksvc env
(restore_ksvc_env on its EXIT trap).

2. The tool tier. Left to their defaults the arms do not match: the Knative arm resolves pods
and gets persistentExecInPod's fast channel, while the VM arm runs over gRPC and has none (§3.1a).
The comparison would then charge the deployment tier for a transport difference, biased against the
VM
. So both arms run the relay + gRPC transportSH_REMOTE_SANDBOX=1 against the in-cluster
relay, which relay-leaf-smoke.sh already exercises. Configuration, not code.

The invariant a reviewer should check: E9 varies the deployment tier and nothing else. Any future arm
inherits both pins.

Preconditions

# Gate Why
1 E8 run first E9 reuses E8's rungs, stub and knee-detection; comparing two tiers before either arm's own knee is understood produces a ratio nobody can attribute
2 #254 merged Same two-arm methodology and the same reporting discipline apply here
3 PR #253 merged Both arms need /turn tool calls actually reaching the sandbox pool
4 Model stub deployable to the cluster It lives at deploy/knative/model-stub/not under deploy/vm/ — precisely because both arms must drive the same stub (§5.4). The VM arm runs the same image locally
5 Same duty basis row on both arms §2.3, named in each run record

Step 7 of P6 §3's build order is exactly this: "E9 second arm on a cluster — both arms against the
same stub and the same gRPC transport."

Acceptance

  • Both arms run on the same host class, same workload, same duty-basis row
  • Both arms recorded as running the relay + gRPC transport; neither arm on persistentExecInPod
  • Both arms against the same stub, with the profile recorded and pinned per arm
  • Connections per session recorded on every rung of both arms and required to match across
    arms
    (§3.4) — a mismatch charges one policy for the other's connection budget
  • The reported figure is a deployment-tier comparison, with the model and tool tiers shown pinned
  • Per-arm knee attribution, not just a ratio
  • No cost/resource-seconds claim (out of scope per §7)
  • Results in deploy/vm/EXPERIMENTS.md, pointer from docs/experiment-results.md

Note on P5

Like E8, E9 is measurable single-subject (§3.6). Mixed-subject rungs and the isolation claim need
#239; the comparison itself does not. State which was run.

Related: E8 run, #254, PR #253, #240 (ScaledJob → elastic pod pool, which changes what the Knative
arm is — if #240 lands first, record which deployment model the cluster arm measured).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by checking E8, #254, and PR #253, then inspect deploy/knative/model-stub/, the E6 driver’s restore_ksvc_env handling, and relay-leaf-smoke.sh. Run both deployment arms only after the listed preconditions, recording matching transport, stub, workload, duty basis, connections per session, and per-arm knees. Put results in deploy/vm/EXPERIMENTS.md and link them from docs/experiment-results.md.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, kubernetes
Domain
devops, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.