MemberJunction / MemberJunction/MJ
Live tier harness: retry-driven budget exhaustion wipes out IT56/IT57, and firstPromptMessages still returns a silent []
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 323
Description
Two harness weaknesses found during v5.51.0 release triage (DEPLOYMENT.md Step 4). Neither is a product defect; both cost signal.
---
## 1. Retry cost is not accounted for in the per-test budget
`IT56 - Agent Payload Guards` and `IT57 - Agent Artifact Tools` both **timed out at ~302s** against the 300s `DEFAULT_TEST_TIMEOUT_MS` (`packages/TestingFramework/Engine/src/drivers/BaseTestDriver.ts:33`), reporting `0/9 checks`:
```
✗ agent-payload-guards.PG5: check 'agent-payload-guards.PG5' exceeded the 38907ms remaining run budget (hung check)
✗ agent-payload-guards.PG6: integration run budget exhausted before check 'agent-payload-guards.PG6' started
... PG7, PG8, PG9 likewise
```
Neither test declares `MaxExecutionTimeMS`, so both inherit the 300s default.
**The cause is compounding retries, not slow checks.** `runWithCompliance` retries a non-compliant scenario up to 3x, each attempt a real model round-trip. In one run, 7 checks went non-compliant — roughly 21 billed model calls — consuming the shared per-test budget before the remaining checks could start. The other failing tests in the same suite finished with **116–247s of headroom**, so the budget is only tight where retries fire.
Net effect: 18 checks reported as failures that never executed, and two bundles produce **zero** product signal while looking maximally red.
Reproduced on two consecutive runs. Raising `AGENT_LIVE_SETTLE_MS`/`AGENT_SETTLE_MS` makes it *worse*, since settle time is charged against the same budget.
### Options
- Give IT56/IT57 an explicit `MaxExecutionTimeMS` sized for worst-case retries (9 checks x 3 attempts).
- Or budget **per check** rather than per run, so one retry-heavy check cannot starve its siblings.
- Either way, `integration run budget exhausted before check '' started` should be reported as a distinct status from a genuine check failure — it currently reads identically in the console.
---
## 2. `firstPromptMessages` returns a silent `[]`
`packages/TestingFramework/integration-test-suite/src/checks/agent-live-shared.ts:367-372`:
```ts
export async function firstPromptMessages(runId, user, provider) {
const steps = await getRunSteps(runId, user, provider);
const firstPrompt = steps.find(s => s.StepType === 'Prompt' && s.TargetLogID);
if (!firstPrompt?.TargetLogID) {
return []; // <- indistinguishable from "read succeeded, no messages"
}
...
```
A caller that gets `[]` cannot tell whether the run genuinely had no prompt step or the harness failed to find one. Downstream this surfaces as `expected 1, got 0` with no indication of which.
This is the **same swallow** v5.51's `fix-live-harness-prompt-run-linkage` set out to eliminate — that changeset added `RequireRows` precisely because *"`RunView` does not throw … each helper coalesced that to `[]`, making a SQL error indistinguishable from 'this run made no model calls'"* — but this path was left returning a bare `[]`. The swallow was relocated, not removed.
### Suggested fix
Throw (or use `RequireRows`) naming the run id and what was missing, so a missing `Prompt` step is reported as such rather than as an empty message list.
---
Related: #3390, #3391, #3392 (same triage).
Contributor guide
Research direction
Start with packages/TestingFramework/Engine/src/drivers/BaseTestDriver.ts:33 and the IT56/IT57 definitions; read how runWithCompliance consumes the shared test budget and run DEPLOYMENT.md Step 4 to reproduce the retry-heavy timeout. Then inspect packages/TestingFramework/integration-test-suite/src/checks/agent-live-shared.ts:367-372 and RequireRows usage. Done means retry-heavy runs preserve useful per-check status and missing prompt steps are distinguishable from an intentionally empty message list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100