JetBrains / JetBrains/thinkrail
Workflow testing: next improvements for the headless harness (slice 3 + eval-style enhancements)
- Dominant language
- TypeScript
- Stars
- 467
- Forks
- 37
- Avg merge
- 1d 56m
- Merged PRs (30d)
- 141
Description
## Context
PR #65 landed the headless workflow-test harness (`e2e/workflows/`, `bun run test:workflows`): declarative `defineScenario` tests drive a real in-process pi agent through the production wiring, with a strict verdict model — **an LLM never decides *pass*** (deterministic checks/signals are binding; persona/simulator/watchdog/judge are input- and diagnosis-side only). Contracts: `e2e/workflows/SPEC.md`; per-skill coverage status: the family table in `packages/pi-thinkrail-workflow/skills/SPEC.md`.
Current coverage: the routing suite (all six router/dispatcher classifications) + infra smokes (mid-flow round-trip, transcript continuation). This issue collects the possible next improvements.
## 1. Slice 3 — worker flows end-to-end (already specced)
Scenario definitions only, no new machinery (`packages/pi-thinkrail-workflow/SPEC.md` § Testing):
- [ ] `starting-a-new-project`: full inception interview driven by the **user simulator** + persona answers → `goal-and-requirements.md` grows to `done`
- [ ] `importing-a-codebase`: headless variant of the existing browser `@agent` e2e (derive + minimal interview → first spec graph, `status: draft`)
- [ ] `brainstorming`: full run — request → task-spec → question rounds → promotion; mid-stage entry via transcript fixtures (record mode)
- [ ] Concept-skill adherence checks (`asking-user-questions`, `writing-specs`) — burns down most of the family table's remaining `unverified by use` debt
## 2. Eval-style harness enhancements (Tessl-inspired, dependency-free)
Ideas adapted from Tessl's Task Evals (https://docs.tessl.io/improving-your-skills/evaluate-skill-quality-using-scenarios) — imported as concepts only; running our workflows under an external eval sandbox is a non-fit because the skills are deliberately host-coupled (`ask_user_question`, `spec_*`, `WORKFLOW_RULE`, the `/skill:` seed — see `module-thinkrail-workflow` § Boundary). All of these must preserve the binding-deterministic invariant; A/B deltas and scores stay **advisory analytics**.
- [ ] **Activation vs. outcome tagging** *(first pick)* — tag checks + run-log entries as `activation` (did the right skill load on its own?) vs. `outcome` (did the flow produce the right artifact/behavior?). The routing suite is activation testing; the known "questions bypass the router" gap is precisely an activation failure — this framing classifies such findings instantly (activation → rule/description problem; outcome → skill-body problem).
- [ ] **N-run averaging** *(first pick)* — `repeat: n` on `defineScenario`: run a scenario several times, aggregate judge verdicts + pass rate in the run record before trusting a drift signal (agents are non-deterministic; a single run can mislead).
- [ ] **Baseline A/B variants** — `variants: ["baseline", "with-workflows"]`: run the same scenario with the workflow extension disabled vs. enabled and report the delta. Measures whether `WORKFLOW_RULE` + the skills actually change behavior (the system's value vs. its token cost) — never measured to date.
- [ ] **Categorized, weighted judge rubrics** — replace the flat statement list with weighted checklist items carrying categories (`INTENT` / `MUST_NOT` / `EDGE_CASE`, partial credit) for more diagnosable advisory data in the run log.
## 3. Longer-term: re-arming verify-by-use (meta-rule 14)
The gate stays suspended because runs land only in a gitignored local log (a recorded decision). Directions worth exploring before re-arming it as a done-gate:
- [ ] **Scenario generation from real history** — derive workflow scenarios from actual thinkrail commits/PRs/sessions instead of hand-authoring (Tessl's commit-scenario idea).
- [ ] **Real-use observation** — mine actual agent session logs for workflow activation/adherence (audit-log-eval style), so "observed flowing through it" comes from real use, not only synthetic scenarios.
- [ ] **A durable, shared run record** — revisit the local-only log decision once there's a place worth binding a done-gate to.
## Related (separate task, not this issue)
The routing suite's open finding — **pure questions bypass the root router** (`WORKFLOW_RULE` omits "question" while the router claims it) — is recorded in `packages/pi-thinkrail-workflow/skills/SPEC.md` § Current limitations & gaps and needs its own brainstormed task: it changes the always-on, byte-stable rule, not the test harness.
Contributor guide
Research direction
Start with e2e/workflows/SPEC.md and run `bun run test:workflows`; then compare the coverage table in packages/pi-thinkrail-workflow/skills/SPEC.md with the unchecked items in this issue. Choose one bounded worker-flow or eval enhancement, trace its existing scenario definitions, and consider it done when deterministic checks and run records cover the selected behavior without weakening the binding-pass invariant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bun, typescript
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100