MemberJunction / MemberJunction/MJ

Live tier harness: retry-driven budget exhaustion wipes out IT56/IT57, and firstPromptMessages still returns a silent []

Open
#3,393 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
2d 1h
Merged PRs (30d)
323

Description

Two harness weaknesses found during v5.51.0 release triage (DEPLOYMENT.md Step 4). Neither is a product defect; both cost signal.

---

## 1. Retry cost is not accounted for in the per-test budget

`IT56 - Agent Payload Guards` and `IT57 - Agent Artifact Tools` both **timed out at ~302s** against the 300s `DEFAULT_TEST_TIMEOUT_MS` (`packages/TestingFramework/Engine/src/drivers/BaseTestDriver.ts:33`), reporting `0/9 checks`:

```
✗ agent-payload-guards.PG5: check 'agent-payload-guards.PG5' exceeded the 38907ms remaining run budget (hung check)
✗ agent-payload-guards.PG6: integration run budget exhausted before check 'agent-payload-guards.PG6' started
... PG7, PG8, PG9 likewise
```

Neither test declares `MaxExecutionTimeMS`, so both inherit the 300s default.

**The cause is compounding retries, not slow checks.** `runWithCompliance` retries a non-compliant scenario up to 3x, each attempt a real model round-trip. In one run, 7 checks went non-compliant — roughly 21 billed model calls — consuming the shared per-test budget before the remaining checks could start. The other failing tests in the same suite finished with **116–247s of headroom**, so the budget is only tight where retries fire.

Net effect: 18 checks reported as failures that never executed, and two bundles produce **zero** product signal while looking maximally red.

Reproduced on two consecutive runs. Raising `AGENT_LIVE_SETTLE_MS`/`AGENT_SETTLE_MS` makes it *worse*, since settle time is charged against the same budget.

### Options
- Give IT56/IT57 an explicit `MaxExecutionTimeMS` sized for worst-case retries (9 checks x 3 attempts).
- Or budget **per check** rather than per run, so one retry-heavy check cannot starve its siblings.
- Either way, `integration run budget exhausted before check '' started` should be reported as a distinct status from a genuine check failure — it currently reads identically in the console.

---

## 2. `firstPromptMessages` returns a silent `[]`

`packages/TestingFramework/integration-test-suite/src/checks/agent-live-shared.ts:367-372`:

```ts
export async function firstPromptMessages(runId, user, provider) {
const steps = await getRunSteps(runId, user, provider);
const firstPrompt = steps.find(s => s.StepType === 'Prompt' && s.TargetLogID);
if (!firstPrompt?.TargetLogID) {
return []; // <- indistinguishable from "read succeeded, no messages"
}
...
```

A caller that gets `[]` cannot tell whether the run genuinely had no prompt step or the harness failed to find one. Downstream this surfaces as `expected 1, got 0` with no indication of which.

This is the **same swallow** v5.51's `fix-live-harness-prompt-run-linkage` set out to eliminate — that changeset added `RequireRows` precisely because *"`RunView` does not throw … each helper coalesced that to `[]`, making a SQL error indistinguishable from 'this run made no model calls'"* — but this path was left returning a bare `[]`. The swallow was relocated, not removed.

### Suggested fix
Throw (or use `RequireRows`) naming the run id and what was missing, so a missing `Prompt` step is reported as such rather than as an empty message list.

---

Related: #3390, #3391, #3392 (same triage).

Contributor guide

Open the contributing guide

Research direction

Start with packages/TestingFramework/Engine/src/drivers/BaseTestDriver.ts:33 and the IT56/IT57 definitions; read how runWithCompliance consumes the shared test budget and run DEPLOYMENT.md Step 4 to reproduce the retry-heavy timeout. Then inspect packages/TestingFramework/integration-test-suite/src/checks/agent-live-shared.ts:367-372 and RequireRows usage. Done means retry-heavy runs preserve useful per-check status and missing prompt steps are distinguishable from an intentionally empty message list.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.