rossoctl / rossoctl/serverless-harness
P4: run E11 with a real model in the loop instead of synthetic Execs
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 7
- Avg merge
- 12h 59m
- Merged PRs (30d)
- 71
Description
Summary
E11's density/throughput numbers (knee at c=8, ~39-43 execs/sec) were driven by a synthetic "closed-loop-per-slot" model issuing trivial Execs (true, cat >/dev/null) back-to-back with no think-time gap and no real tool-call payload. This issue is the full validation experiment: replace the synthetic driver with a real agent loop (real model calls, real tool calls) and see whether the throughput/latency picture — and the "sandbox overhead hides behind think-time" argument — survives contact with realistic traffic.
Why this matters
- This is the deck's own flagged gap ("No model in the loop... the arithmetic of hiding replenishment behind thinking time is argued, not measured"), and it's the most expensive item in this epic to stand up, so it should come after the cheaper, more targeted items (#257 think-time distribution, #258 resume/destroy decomposition) rather than before them — those inform what this experiment should even look for.
- Everything currently measured (E10's 52.74 ms, E11's throughput ladder) is a mechanism measurement, isolated from real workload shape. This is what turns it into a service-level measurement.
- It also tests whether the current synthetic-Exec shape (near-instant commands) has been hiding any latency that real tool calls (longer-running commands, larger outputs, more filesystem activity) would expose — e.g., mount-at-acquire cost, or Σ PSS growth under real memory usage rather than idle standbys.
What's already known / where to look
- Driver:
deploy/microvm/e11-density.sh,drivingModel: "closed-loop-per-slot"— the code path that would need a real alternative. The script already supports a model-stub hook:SH_E11_MODEL_STUB_CMD(seedeploy/microvm/EXPERIMENTS.md's "Model-stub dependency gap" note — "absent one it drives the mix itself directly, a disclosed stand-in for the real stub, not the stub itself"). - The harness (
harness/) and Pi coding agent (pi-fork/) already run real agent sessions elsewhere in the repo; the P6 density work used a model stub with a declared latency profile (300ms TTFT, 12ms/token) rather than a live model — a middle ground between fully synthetic and fully real that's worth considering as a cheaper intermediate step. - Open item from
EXPERIMENTS.md's "what happens next" list: "Live gate at low concurrency — validation, not a headline number... blocked on: a model key and a deployed relay." This issue is effectively that item, scaled up from a validation gate to a full density re-run.
Open questions for the spec
- Real model (which one, at what concurrency-affecting cost — e.g., a hosted API's own rate limits could become a confound), or a declared-latency stub calibrated to #257's measured think-time distribution (cheaper, and directly answers "does the mechanism hold under a realistic gap" without needing a live model dependency)? I'd lean toward the stub as a first cut, gated on #257 actually producing a distribution to calibrate it against.
- Should this reuse the existing E11 ladder shape (sweep c, look for a knee), or does a real workload's arrival pattern (bursty, not closed-loop) need a different driving model entirely?
- What tool-call mix is representative — the repo already has a git-chain-based tool workload from the P6 work that could be reused rather than invented from scratch.
Suggested scope for a first cut
Depends on #257 landing first (calibration data). Once available: repeat the c=1,2,4,8 ladder with a declared-latency model stub matching the measured think-time distribution, and check whether the knee position and throughput ceiling change materially from the fully-synthetic baseline.
Part of #256 (P4 epic: next experiments after the metal STOP).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with deploy/microvm/e11-density.sh and the related sections of deploy/microvm/EXPERIMENTS.md, then review #257's think-time distribution before choosing a driver. Re-run the c=1,2,4,8 ladder with a declared-latency model stub or real model and representative tool calls; done means comparing the knee and throughput ceiling with the synthetic baseline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- ai, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100