rossoctl / rossoctl/serverless-harness

E8: run VM density + saturation (both arms) and record the result

Open
#275 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
7
Avg merge
12h 59m
Merged PRs (30d)
71

Description

Summary

Run E8 — VM density and saturation (P6 §5.2) — and record the result. The spec amendment and driver
work are #254; this issue is the run itself, which #254 explicitly does not cover.

Splitting them is deliberate. #254 lands a two-arm methodology (capacity + realism) and the driver
changes that make a knee reachable; it can be reviewed and merged without any hardware booked. The run
consumes that methodology, produces the number, and is where a bound=not-observed outcome has to be
adjudicated rather than merged. Keeping them in one issue is what let the previous calibrated run be
recorded as a density figure when it was stub-sleep-bound.

Preconditions — all of them, checked before booking anything

# Gate Why it blocks
1 #254 merged Without the capacity arm the ladder tops out at the admission cap W×S=32, a config choice, and the run reproduces bound=not-observed
2 PR #253 merged /turn's tool calls must reach the sandbox pool. A run where tools execute locally measures a harness with no hands tier — and the counting probe is what catches it
3 #239 merged if mixed-subject rungs are claimed P6 §3.6/§7: single-subject rungs exercise every mechanism this slice adds and need no P5. Mixed subjects and the isolation claim need it. Decide which run this is before starting
4 Model stub deployed with a tool-call profile (§5.4) A text-only stub means no session reaches the sandbox and the density number silently excludes the hands tier
5 Duty basis pinned to one row of §2.3 Default E6/OCP: duty = 0.061–0.079, provision at 12.6–16.5. The driver asserts K ≥ ceil(W × S × duty) with duty and the ratio from the same row

Gate 3 is a scoping decision, not a dependency to wait on. Per P6 §3.6, P5 slipping delays a claim,
not the work — so a single-subject E8 is runnable the moment gates 1, 2, 4, 5 hold, and should not be
held behind #239.

What to run

Both arms from #254's amended §5.2, each recorded separately and never merged into one figure:

  • capacity arm — trivial exec, fast stub profile, sandbox tier non-binding and non-competing.
    Sweep offered concurrency upward with SH_WORKERS × SH_TURNS_PER_WORKER and
    KAGENTI_SANDBOX_CAP set well past the expected knee. Reports an upper bound on the harness
    tier
    — explicitly not a deployable density.
  • realism arm — calibrated duty (e6-ocp basis), stub profile 300/12/64. This is the deployable
    number, sandbox-shaped by construction.

W=1 is a rung, not an aside: it makes "one process would be simpler" a data point (§5.2).

Knee detection reuses E6's sustained-decline detectKnee with degradeX=2 against a warm C=1
baseline, and the knee is reported as a floor, not a ceiling (EXPERIMENTS.md:120).

Acceptance

  • Both arms run; each arm's stub profile recorded and pinned via assert_stub_pinned, and the
    arm label verified against what /profile returns
  • Per rung: throughput, p50/p95, event-loop lag p99, RSS, per-turn worker CPU, sandbox-container
    CPU, lease saturation, over-admission events, spurious 429s
  • Duty basis row and connections per session recorded on every rung, not only sticky ones
  • The knee is attributed — worker CPU vs RSS vs replenishment vs admission cap — not just located
  • If the capacity arm still returns bound=not-observed, say so and name what bound first; do not
    report a floor as a ceiling
  • % of ideal throughput is not reported as a capacity figure when the ideal is stub-sleep-bound
  • Single-subject vs mixed-subject scope stated in the record, with the isolation claim made only
    if #239 landed
  • Results in deploy/vm/EXPERIMENTS.md, pointer added from docs/experiment-results.md (§5.6)

Open question to settle during the run

Is event-loop lag trustworthy as the capacity arm's headline diagnostic? It read ~11 ms flat at
every rung including c=1, where the tier is nearly idle — which looks like a sampling or timer
artefact. §5.2 makes it the metric that attributes a knee to worker CPU, so verify it against a
deliberately CPU-starved worker before relying on it. If it cannot be trusted, per-turn worker CPU
(#254 §2e) carries the attribution alone and the record should say so.

Related: #254 (spec + driver), PR #253 (routing prerequisite), #239 (mixed-subject scope), E9 run.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

First verify gates #254, PR #253, the stub profile, and the pinned §2.3 duty row before booking the run. Use the amended §5.2 procedure for separate capacity and realism arms, then record per-rung metrics and attribution in deploy/vm/EXPERIMENTS.md and add the pointer to docs/experiment-results.md.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
infrastructure, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.