MemberJunction / MemberJunction/MJ

prompt-eval-harness PE4 and PE7 skip on every PR, so the bundle's two end-to-end checks evaluate nothing in CI

Open
#4,438 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TSQL
Stars
29
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
308

Description

## Summary

The `prompt-eval-harness` bundle exists to prove — deterministically, per PR, for zero tokens — that the eval harness measures what it claims to. Two of its seven checks, **PE4** and **PE7**, cannot run on a CI runner. As of #4176 they skip-as-pass rather than fail, so the lane is green while those two checks evaluate nothing.

This is deliberate and the right short-term behaviour (failing a lane over a fixture it was never going to have is worse), but it leaves a real coverage gap that should be closed or consciously accepted.

## Why each one skips

They skip for **different** reasons, and only one is worth fixing.

### PE4 — needs credentials, and `TestLLM` registration is not enough to avoid them

PE4 registers a `TestLLM` over `['OpenAILLM', 'AnthropicLLM', 'GeminiLLM', 'CerebrasLLM', 'GroqLLM']` and runs the real `AIPromptRunner`, on the assumption that stubbing the driver classes makes the run offline.

It doesn't. **Model selection filters candidates on configured API credentials _before_ the `ClassFactory` ever instantiates a driver.** With no keys present, every candidate is rejected and the run fails at selection — the registered `TestLLM` is never reached. The check's own next assertion (`llm.CalledModels.length > 0`) is written to catch exactly that, but execution never gets there.

Observed on a CI runner:

```
No suitable model found for prompt SEO AIEO Specialist Agent - Main Prompt.
No valid API credentials/keys are configured for any of the candidate model-vendor combinations.
Tried: Gemini 3.1 Flash-Lite/Google, Gemini 3.1 Flash-Lite/Vertex AI, GPT-OSS-120B/Cerebras,
GPT 5.5/OpenAI, Kimi K2.5/OpenRouter (2020 total).
```

Two secondary problems visible in that message:

1. **The prompt is non-deterministic.** PE4 selects `AIEngine.Instance.Prompts.find(p => p.Status === 'Active')` — whichever active prompt happens to come first. It differs per database; CI landed on `SEO AIEO Specialist Agent - Main Prompt`.
2. **The candidate list escapes the stub.** `Vertex AI` and `OpenRouter` are not in the `registerTestLLM` list. If credentials for those ever existed, PE4 would instantiate the **real** drivers and make **real, billed API calls** from a lane documented as `NO REAL LLM CALLS`.

### PE7 — needs the corpus in the database, which CI deliberately does not push

PE7 queries `MJ: Tests` for a record the generator produced. Those records live in `metadata-optional/prompt-eval-corpus/tests`. CI pushes only two directories (`.github/workflows/integration.yml`):

```
sync push --dir=metadata
sync push --dir=metadata-optional/integration-test
```

The corpus is a **measurement fixture, not product metadata**, and the decision is that it stays out of CI. So PE7 has nothing to drive there, by design.

**PE7 needs no fix** — skip-as-pass is the correct end state. It still provides real coverage wherever the corpus *is* pushed (a developer database, an eval host), and that is where the generator↔driver contract it guards actually matters. It is listed here only so the coverage gap is recorded rather than discovered again later.

## Why this matters

PE7's own history is the argument. Its comment records two shipped bugs it was written to catch, both invisible to a green unit suite:

- the generator wrote `agentName` while `PromptEvalConfig` read only `agentId`, so all 171 records failed before reaching a model;
- `AgentDecisionOracle` called `evaluateDecision` instead of `evaluateCorpusExpectation`, so all 13 `anyOf` cases scored a failure no model could have avoided.

Both are shape mismatches between components that unit-test cleanly in isolation. That class of bug is precisely what PE4 and PE7 exist to catch, and on a PR neither is currently looking.

Their green should be read as **"not evaluated"**, not "passed". PE1–PE3, PE5 and PE6 never skip and carry the bundle's real per-PR weight.

## Proposed fix (PE4 only)

Add an AI credential fixture for the stubbed vendors to `metadata-optional/integration-test/` — the directory CI **does** push. Model selection then finds a usable candidate, hands off to the registered `TestLLM`, and PE4 runs offline in CI as originally designed. The key value is never used for anything: `TestLLM` intercepts at the `ClassFactory` before any network call.

Worth pairing with two hardening changes:

- **Pin PE4's prompt** to a known fixture instead of `find(p => p.Status === 'Active')`, so the check tests the same thing on every database.
- **Assert the candidate set is fully stubbed** — fail loudly if selection offers a driver class not in the `registerTestLLM` list, rather than silently making a real API call.

## Current behaviour after #4176

Both checks log a specific reason and return:

```
→ skipped: no model-vendor in this database has configured credentials
(selection rejects every candidate before TestLLM is reached)

→ skipped: '' is not in this database
(corpus not pushed — mj sync push --dir=metadata-optional/prompt-eval-corpus)
```

The bundle header in `packages/TestingFramework/integration-test-suite/src/checks/prompt-eval-harness.checks.ts` documents which checks gate a PR and which do not.

## Acceptance

- [ ] PE4 executes its assertions on a credential-less CI runner (does not skip)
- [ ] PE4 drives a pinned prompt rather than an arbitrary active one
- [ ] PE4 fails loudly if model selection offers an un-stubbed driver class
- [ ] PE7's skip remains, documented as intended behaviour

Contributor guide

Open the contributing guide

Research direction

Start with packages/TestingFramework/integration-test-suite/src/checks/prompt-eval-harness.checks.ts and the CI pushes in .github/workflows/integration.yml. Trace PE4 through AIPromptRunner, model selection, and registerTestLLM, then inspect metadata-optional/integration-test for the credential fixture. Done means PE4 runs its assertions on CI, uses a pinned prompt, rejects un-stubbed drivers, and PE7 remains an intentional skip.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.