Add a hostile certification fixture: prove the runtime probes report on the product, not on whatever else is running

Why this exists

Almost every runtime-gate defect is an attribution defect — the probe reporting confidently about something that is not the product — and attribution can only go wrong when there is something else present to be confused with the product.

That sentence is the whole justification for this issue. Without it, the fixture list below reads as six oddly-specific pieces of paranoia. With it, they are the minimum.

The evidence is concrete. A code review of the runtime gates (#1719) found six attribution defects, and every one of them passed 55/55 grader certification. They surfaced only under adversarial conditions — a process squatting on the port, an app ignoring SIGTERM, a ternary in a request body — none of which the golden fixture contains.

The worst of them turned a gate green on a crashing app: a process holding the fixture's declared port answered before the child could bind and was accepted as the app, so the crash-on-boot mutation reported no issues, while the golden health case reported healthEndpointUnhealthy — a stranger's 404 billed to the agent as a product failure. Both halves of the exit-code contract violated at once, at 55/55.

The general lesson, stated plainly

A green certification number is evidence that a probe agrees with itself on a clean machine. It becomes evidence that a probe is correct only when something adversarial is present.

That is the difference between the number we have and the number we want. It is also precisely the property the gate in #1669 lacked while reporting healthy — it was 0-for-16 across every run ever executed, because its probe signed requests with a corrupted key, and nothing in the certification set could tell.

This is not "a fixture with a broken app". Mutations already cover broken apps, and they work. This is a hostile environment, plus a few input shapes that break parsers.

---

⚠️ The squatter must bind 0.0.0.0, not 127.0.0.1

Calling this out on its own because it is the highest-value detail in this issue and the one most likely to be lost in implementation.

When the port-squatter bug was first fixed, the fix added a free-port guard that bound 127.0.0.1. The squatter held 0.0.0.0. macOS reported the port free anyway, and the bug survived completely unchanged — the free check and the readiness check were asking different questions. It was only caught because the reproduction used a squatter bound to 0.0.0.0.

A fixture whose squatter binds 127.0.0.1 looks hostile, passes, and would have certified the broken fix.

That is worse than having no hostile fixture at all, because it converts "we have not tested this" into "we tested this and it is fine" — the same class of failure as everything else described here.

The same asymmetry applies to the allocator, not just the checker

The spec above would not have caught the second instance of this bug, and that is worth fixing in the spec rather than discovering later.

After isPortFree was corrected to bind 0.0.0.0, findFreePort was still asking the OS for a free ephemeral port on 127.0.0.1. Same wrong question, one layer up: the allocator could hand back a port a squatter already held on all interfaces, the app would fail to bind it, and readiness would connect to the squatter and report provenance: 'remapped' on the first tick.

A squatter on a fixed declared port never exercises findFreePort. To cover the allocation path, the fixture needs a squatter on a port the allocator might return — a spread across the ephemeral range, or a hook that squats whatever port was just allocated — asserting that a port handed out by the allocator is never one somebody else holds.

This one is lower probability than the fixed-port case, because the OS usually picks a genuinely unused port. That is precisely why it belongs here: an intermittent attribution failure gets written off as flake rather than investigated.

---

What the fixture needs, and the defect each item catches

0. Fixtures that differ in form but not in meaning

The most productive class, added after it found two real defects, and the cheapest to write.

Take the golden fixture and change only the incidental syntax of something a gate parses — same routes, same behaviour, same verdict expected. Then assert the verdict does not move.

The case that found it: fetch('/api/items') → fetch(`${base}/items`). Identical calls, identical routes, one template literal — and both runtime-frontend-api and runtime-crud stood down silently, the first being the gate that exists to catch a frontend and backend that were never connected. An env-configured API base is arguably the modal way a real scaffolded frontend is written.

Why this class is needed at all, and it generalises past this repo: certification fixtures written by the person who wrote the parser will use the syntax the parser handles. Not carelessness — you cannot write a fixture in a syntax you did not think of. Every mutation is drawn from the same imagination as the code it tests, which is why 84/84 kept agreeing with a parser that could not read ordinary JavaScript.

Worth covering, at minimum:

• computed URL bases (template literals, string concatenation, axios.create({ baseURL }))
• a client wrapper (api.get('/items')) rather than a bare fetch
• request bodies built by a helper rather than an inline object literal
• a health path declared only in one of the three artifacts that may declare it
• the same start command expressed via runtimeExecutable rather than program

Each is a one-line edit to a fixture and each asserts no change in verdict. Fixed in #1730 for the first case; the rest are unguarded.

---

1. A port squatter — highest value, this one turned a gate green

A process holding the port the fixture declares (7071, the Azure Functions default and therefore a realistic collision), bound to 0.0.0.0, answering 404.

Caught: the declared port was polled from the first readiness tick, before the child could possibly have bound anything, so the squatter answered first and was accepted as the app.

Assertions: golden gates still pass; every mutation still goes red; no verdict cites the squatter's responses.

Also assert: a child that dies of EADDRINUSE is never reported as listening. Readiness must check child liveness before probing sockets — checked afterwards, a dead process whose port a squatter answers still reads as healthy.

2. An app that ignores SIGTERM, started via npm start

A shell parent forking a node child, with a SIGTERM handler that logs and does nothing.

Caught: teardown that signals only the pid it was handed leaves the grandchild holding the port. Not a wrong verdict — a leak, which breaks the next run for a reason that looks nothing like its cause.

Needs both properties: the intermediate shell, so the process-group kill matters, and the ignored signal, so the SIGKILL escalation matters.

Assertion: after the run, the port accepts no connections and no descendant survives.

3. A devDependencies-only project with no node_modules

Start script invoking vite / tsx / nodemon, those declared in devDependencies, nothing installed.

Caught: the "not installed yet" guard read dependencies only, so it never fired; npm start hit "command not found"; the app was reported as a product failure. Exactly the misattribution that guard exists to prevent.

Assertion: exit 3, not exit 1.

4. Request bodies that break the extractor

In the served frontend's POST call:

• a ternary — { name: input.value, done: isDone ? true : false }
• a regex literal — { name: url.replace(/['"]/g, '') }

Caught: the ternary's top-level : was read as a second field name, so the probe posted a field the API never declared and a correctly-implemented API rejected it with 400 — a fabricated exit 1. The regex's quote desynchronised the bracket scanner, so the CRUD gate silently degraded to "no collection route" and stopped testing anything.

Assertions: extracted fields are exactly name/done; the collection route is still found with the regex present.

5. Noisy startup output naming a datastore port

Log lines emitted before listening: Connecting to postgres at db.local port: 5432, [db] pool created { host: 'localhost', port: 5432 }.

Caught: a bare port: pattern matched connection logging emitted before the app listened. On a machine with a local Postgres running, the harness would have connected to the database and called it the application.

Stronger with a listener on 5432 — but that needs a datastore, so a plain socket on 5432 answering nothing is the containerless version.

6. A background worker, and a worker that dies

Two fixtures, and the pair is the point.

A stack declaring project.api: none is a background worker with no HTTP surface. runtime-app-starts asserts it is still running after a settling window rather than listening — a positive assertion, deliberately not an exemption, because "this stack declares no API, so skip the check" would be a gate that is wired, runs, and cannot fail, and one that fails open on a declaration, so a mis-declared stack would buy itself immunity.

• healthy worker — entrypoint stays alive. Asserts the gate passes, and that gates 2-5 report noHttpSurface.
• dying worker — entrypoint returns immediately. Must go red with appExitedBeforeReadiness specifically, not merely red.

The second assertion matters more than it looks. A fixture that exits instantly can fail for at least three unrelated reasons — no start command resolved, dependencies not installed, spawn failure — and two of those exit 3. Checking only that certification is green would certify a mutation that never reaches the branch it names. That trap was hit for real on the template-literal case in #1730, where the first version defeated one of two call sites, read as 83/84, and argued for its own weakening.

Why this needs the setup machinery. Both fixtures require a stack declaration (api: none) to be visible to the grader, which today means a projection at /agent/assets/stack.json. That is per-fixture environment setup, the same hook the port squatter needs. The alternative — a workspace-scoped override path in readStackProjection — was considered and rejected: a test-only lookup path in production code is invisible to every other reader and becomes load-bearing the moment someone assumes it is supported. --require-health was exactly that shape.

Verified by hand in both directions in #1734; uncertified pending this issue.

---

7. An app that hard-codes its port and ignores PORT

Assertion: still located (via its own announced output), and the portEnvironmentVariableIgnored finding fires — but only when the variable was genuinely passed. A start script with an inline PORT=3000 overrides the child environment, so claiming the app "ignored PORT" in that case is a fabricated statement about the product.

---

Shape, and the one real infrastructure change

Certification mutations are single-file text edits, which cannot express "start a squatter first". This needs a fixture-level setup/teardown hook — the smallest useful version is optional setup/teardown scripts a fixture may declare, run around its validators, with teardown guaranteed even on failure.

That hook is the only genuine infrastructure change here. Items 3–6 are just fixture content.

Sequencing — please read before starting

The hook touches evals/src/graderCertification.ts, which is under concurrent edit:

• It must land after the fidelity-gates changes to that file (#1721), in its own PR, touching only the harness.
• Three sessions editing that one file concurrently is what caused avoidable rebase pain already.

Whoever picks this up should confirm #1721 has landed first, so they do not start by rebasing on sand.

---

What it buys

Today, a green certification number means the probe agrees with itself on a clean fixture.

With this, it starts to mean the probe reports on the product and not on whatever else happens to be running — which is the property the runtime gates actually need in order for their verdicts to be worth anything.

Context: #1719 (the runtime gates), #1669 (the 0-for-16 gate that motivated gate certification in the first place).

After #1721 lands, read evals/src/graderCertification.ts and trace the existing certification fixture flow. Check how /agent/assets/stack.json is projected for worker fixtures, then review the listed hostile cases and their assertions. Done means setup/teardown is guaranteed around validation and the fixtures distinguish attribution failures, leaks, missing dependencies, parser failures, and worker exit causes.