MemberJunction / MemberJunction/MJ
prompt-eval-harness PE4 and PE7 skip on every PR, so the bundle's two end-to-end checks evaluate nothing in CI
- Dominant language
- TSQL
- Stars
- 29
- Forks
- 6
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 308
Description
## Summary
The `prompt-eval-harness` bundle exists to prove — deterministically, per PR, for zero tokens — that the eval harness measures what it claims to. Two of its seven checks, **PE4** and **PE7**, cannot run on a CI runner. As of #4176 they skip-as-pass rather than fail, so the lane is green while those two checks evaluate nothing.
This is deliberate and the right short-term behaviour (failing a lane over a fixture it was never going to have is worse), but it leaves a real coverage gap that should be closed or consciously accepted.
## Why each one skips
They skip for **different** reasons, and only one is worth fixing.
### PE4 — needs credentials, and `TestLLM` registration is not enough to avoid them
PE4 registers a `TestLLM` over `['OpenAILLM', 'AnthropicLLM', 'GeminiLLM', 'CerebrasLLM', 'GroqLLM']` and runs the real `AIPromptRunner`, on the assumption that stubbing the driver classes makes the run offline.
It doesn't. **Model selection filters candidates on configured API credentials _before_ the `ClassFactory` ever instantiates a driver.** With no keys present, every candidate is rejected and the run fails at selection — the registered `TestLLM` is never reached. The check's own next assertion (`llm.CalledModels.length > 0`) is written to catch exactly that, but execution never gets there.
Observed on a CI runner:
```
No suitable model found for prompt SEO AIEO Specialist Agent - Main Prompt.
No valid API credentials/keys are configured for any of the candidate model-vendor combinations.
Tried: Gemini 3.1 Flash-Lite/Google, Gemini 3.1 Flash-Lite/Vertex AI, GPT-OSS-120B/Cerebras,
GPT 5.5/OpenAI, Kimi K2.5/OpenRouter (2020 total).
```
Two secondary problems visible in that message:
1. **The prompt is non-deterministic.** PE4 selects `AIEngine.Instance.Prompts.find(p => p.Status === 'Active')` — whichever active prompt happens to come first. It differs per database; CI landed on `SEO AIEO Specialist Agent - Main Prompt`.
2. **The candidate list escapes the stub.** `Vertex AI` and `OpenRouter` are not in the `registerTestLLM` list. If credentials for those ever existed, PE4 would instantiate the **real** drivers and make **real, billed API calls** from a lane documented as `NO REAL LLM CALLS`.
### PE7 — needs the corpus in the database, which CI deliberately does not push
PE7 queries `MJ: Tests` for a record the generator produced. Those records live in `metadata-optional/prompt-eval-corpus/tests`. CI pushes only two directories (`.github/workflows/integration.yml`):
```
sync push --dir=metadata
sync push --dir=metadata-optional/integration-test
```
The corpus is a **measurement fixture, not product metadata**, and the decision is that it stays out of CI. So PE7 has nothing to drive there, by design.
**PE7 needs no fix** — skip-as-pass is the correct end state. It still provides real coverage wherever the corpus *is* pushed (a developer database, an eval host), and that is where the generator↔driver contract it guards actually matters. It is listed here only so the coverage gap is recorded rather than discovered again later.
## Why this matters
PE7's own history is the argument. Its comment records two shipped bugs it was written to catch, both invisible to a green unit suite:
- the generator wrote `agentName` while `PromptEvalConfig` read only `agentId`, so all 171 records failed before reaching a model;
- `AgentDecisionOracle` called `evaluateDecision` instead of `evaluateCorpusExpectation`, so all 13 `anyOf` cases scored a failure no model could have avoided.
Both are shape mismatches between components that unit-test cleanly in isolation. That class of bug is precisely what PE4 and PE7 exist to catch, and on a PR neither is currently looking.
Their green should be read as **"not evaluated"**, not "passed". PE1–PE3, PE5 and PE6 never skip and carry the bundle's real per-PR weight.
## Proposed fix (PE4 only)
Add an AI credential fixture for the stubbed vendors to `metadata-optional/integration-test/` — the directory CI **does** push. Model selection then finds a usable candidate, hands off to the registered `TestLLM`, and PE4 runs offline in CI as originally designed. The key value is never used for anything: `TestLLM` intercepts at the `ClassFactory` before any network call.
Worth pairing with two hardening changes:
- **Pin PE4's prompt** to a known fixture instead of `find(p => p.Status === 'Active')`, so the check tests the same thing on every database.
- **Assert the candidate set is fully stubbed** — fail loudly if selection offers a driver class not in the `registerTestLLM` list, rather than silently making a real API call.
## Current behaviour after #4176
Both checks log a specific reason and return:
```
→ skipped: no model-vendor in this database has configured credentials
(selection rejects every candidate before TestLLM is reached)
→ skipped: '' is not in this database
(corpus not pushed — mj sync push --dir=metadata-optional/prompt-eval-corpus)
```
The bundle header in `packages/TestingFramework/integration-test-suite/src/checks/prompt-eval-harness.checks.ts` documents which checks gate a PR and which do not.
## Acceptance
- [ ] PE4 executes its assertions on a credential-less CI runner (does not skip)
- [ ] PE4 drives a pinned prompt rather than an arbitrary active one
- [ ] PE4 fails loudly if model selection offers an un-stubbed driver class
- [ ] PE7's skip remains, documented as intended behaviour
Contributor guide
Research direction
Start with packages/TestingFramework/integration-test-suite/src/checks/prompt-eval-harness.checks.ts and the CI pushes in .github/workflows/integration.yml. Trace PE4 through AIPromptRunner, model selection, and registerTestLLM, then inspect metadata-optional/integration-test for the credential fixture. Done means PE4 runs its assertions on CI, uses a pinned prompt, rejects un-stubbed drivers, and PE7 remains an intentional skip.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100