rossoctl / rossoctl/serverless-harness

P4: run E11 with a real model in the loop instead of synthetic Execs

Open
#261 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
7
Avg merge
12h 59m
Merged PRs (30d)
71

Description

Summary

E11's density/throughput numbers (knee at c=8, ~39-43 execs/sec) were driven by a synthetic "closed-loop-per-slot" model issuing trivial Execs (true, cat >/dev/null) back-to-back with no think-time gap and no real tool-call payload. This issue is the full validation experiment: replace the synthetic driver with a real agent loop (real model calls, real tool calls) and see whether the throughput/latency picture — and the "sandbox overhead hides behind think-time" argument — survives contact with realistic traffic.

Why this matters

  • This is the deck's own flagged gap ("No model in the loop... the arithmetic of hiding replenishment behind thinking time is argued, not measured"), and it's the most expensive item in this epic to stand up, so it should come after the cheaper, more targeted items (#257 think-time distribution, #258 resume/destroy decomposition) rather than before them — those inform what this experiment should even look for.
  • Everything currently measured (E10's 52.74 ms, E11's throughput ladder) is a mechanism measurement, isolated from real workload shape. This is what turns it into a service-level measurement.
  • It also tests whether the current synthetic-Exec shape (near-instant commands) has been hiding any latency that real tool calls (longer-running commands, larger outputs, more filesystem activity) would expose — e.g., mount-at-acquire cost, or Σ PSS growth under real memory usage rather than idle standbys.

What's already known / where to look

  • Driver: deploy/microvm/e11-density.sh, drivingModel: "closed-loop-per-slot" — the code path that would need a real alternative. The script already supports a model-stub hook: SH_E11_MODEL_STUB_CMD (see deploy/microvm/EXPERIMENTS.md's "Model-stub dependency gap" note — "absent one it drives the mix itself directly, a disclosed stand-in for the real stub, not the stub itself").
  • The harness (harness/) and Pi coding agent (pi-fork/) already run real agent sessions elsewhere in the repo; the P6 density work used a model stub with a declared latency profile (300ms TTFT, 12ms/token) rather than a live model — a middle ground between fully synthetic and fully real that's worth considering as a cheaper intermediate step.
  • Open item from EXPERIMENTS.md's "what happens next" list: "Live gate at low concurrency — validation, not a headline number... blocked on: a model key and a deployed relay." This issue is effectively that item, scaled up from a validation gate to a full density re-run.

Open questions for the spec

  1. Real model (which one, at what concurrency-affecting cost — e.g., a hosted API's own rate limits could become a confound), or a declared-latency stub calibrated to #257's measured think-time distribution (cheaper, and directly answers "does the mechanism hold under a realistic gap" without needing a live model dependency)? I'd lean toward the stub as a first cut, gated on #257 actually producing a distribution to calibrate it against.
  2. Should this reuse the existing E11 ladder shape (sweep c, look for a knee), or does a real workload's arrival pattern (bursty, not closed-loop) need a different driving model entirely?
  3. What tool-call mix is representative — the repo already has a git-chain-based tool workload from the P6 work that could be reused rather than invented from scratch.

Suggested scope for a first cut

Depends on #257 landing first (calibration data). Once available: repeat the c=1,2,4,8 ladder with a declared-latency model stub matching the measured think-time distribution, and check whether the knee position and throughput ceiling change materially from the fully-synthetic baseline.


Part of #256 (P4 epic: next experiments after the metal STOP).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with deploy/microvm/e11-density.sh and the related sections of deploy/microvm/EXPERIMENTS.md, then review #257's think-time distribution before choosing a driver. Re-run the c=1,2,4,8 ladder with a declared-latency model stub or real model and representative tool calls; done means comparing the knee and throughput ceiling with the synthetic baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
ai, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.