rossoctl / rossoctl/serverless-harness

E13: root-cause E12's rung C@128 dropped connections — and the CPU-contention follow-up it opened

Open
#280 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1
Forks
7
Avg merge
12h 59m
Merged PRs (30d)
71

Description

Why this issue exists

E13 ran and returned a verdict, but it lives only as PR #273 — so it cannot be a sub-issue of the
epic (rossoctl/rossoctl#2546) and does not appear in any rollup, even though #277 (P4.1 implementation)
names it as a blocking gate. This issue is that gate, plus the follow-up E13 explicitly deferred.

Same pattern as E12, which has both an issue (#271) and a PR (#272). E13 was missing its issue half.

What E13 already settled (do not re-run)

Two diagnostics on the real nested-m8i rig, both in PR #273:

Diagnostic Result
No-second-port control (e13-restore-capacity-control.sh) — N=128 concurrent restores using only the already-shipped agent Exec path (port 1024), via E12's E12_PROBE_SOURCE_ONLY=1 hook Clean, 0/128
Memory/OOM telemetry (e13-mem-telemetry.sh + e13-correlate.py) — samples MemAvailable, watches OOM killer around a rerun of E12's rung C@128 E12's rung still failed 38/128 with the identical read: EOF signature, but zero OOM events, memory never below ~12.3 GiB of 15 GiB

Sealed prediction 7 (memory exhaustion) is falsified on both letter and substance.

Two of the 38 failures show host_witness:"yes" — the guest→host leg of the 1025 round trip
demonstrably completed before the unrelated 1024 relay died. Direct, if partial, evidence.

T1 (route sandbox egress over vsock) stands, on firmer ground than after E12.

The open question this issue carries

E13's conclusion is an inference by elimination, not a direct measurement: the evidence points to
CPU/scheduling contention from the added workload E12's probe mechanism introduces — an extra
host-side python3 listener per VM, plus a heavier per-VM guest Exec payload. That was stated honestly
as an inference in #273, with a follow-up named rather than performed.

So P4.1's scale question is re-scoped, not answered:

  • "how many concurrent restores can we do?" — answered: not the lever, at least at N=128
  • "how much concurrent CPU-bound work runs alongside a restore burst, and does the Exec channel stay
    healthy under it?"
    — open

That reframing matters for #277: the thing to design against is contention from neighbouring work, not
a restore-count ceiling.

Acceptance

  • Confirm the CPU/scheduling reading with a direct measurement rather than elimination —
    isolate the per-VM host listener and the heavier guest payload as separate variables, since E13
    could not distinguish them
  • Characterise Exec-channel health under concurrent CPU-bound load alongside a restore burst
  • Decide whether the read: EOF signature on the 1024 relay is a P4.1 concern or a pre-existing
    Exec-path fragility that E12's probe merely exposed — these have different owners
  • Record whether the recommendations in #273's write-up change anything in #277's scope
  • Results appended to deploy/microvm/EXPERIMENTS.md ### E13

Substrate note

nested-m8i, not metal — consistent with E12 (#271) and for the same reason: the probe rootfs differs
from the authoritative E10/E11 snapshot digest (sha256:668af5893e9c70ef), and reusing metal's
snapshot would invalidate that comparison invisibly. e10-lifecycle.sh prints a STOP/MANDATORY verdict
only when SH_SUBSTRATE is exactly "metal", so nothing here claims metal fidelity.

Related: PR #273 (the run), #271 / PR #272 (E12), #277 (P4.1 implementation — gated on this), PR #268
(P4.1 spec), #266 (nested/metal ratio, supplies substrate).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read PR #273 and the E13 section of deploy/microvm/EXPERIMENTS.md first. Use e13-restore-capacity-control.sh, e13-mem-telemetry.sh, and e13-correlate.py as the existing diagnostic entry points, then design measurements that separate the host listener from the guest payload and exercise CPU-bound load beside restores. Done means the contention source and Exec-channel behavior are measured, ownership of the EOF failure is decided, and results are appended under E13.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
distributed-systems, infrastructure, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.