Build a Pi-backed black-box Pipeline evaluation harness

> 🤖 Drafted by Codex (AI assistant) on behalf of jmchilton.

## Problem

The current `/test-pipeline` path is useful for exploratory refinement, but it is too fuzzy to serve as a convincing conformance test for published Pipeline and skill artifacts.

Today it:

- delegates to `/test-drive` and manually walks Pipeline frontmatter rather than invoking the published `pipeline-*` harness;
- bypasses runtime skill dispatch by reading each cast `SKILL.md` directly;
- lets one agent orchestrate, execute, retain every phase's context, and evaluate the result;
- explicitly permits inputs from prior `_emulated-runs`, which are visible inside the checkout;
- does not enforce the assembled harness's optional `--use-subagents` contract;
- applies `eval.md` assertions from prose, with the same agent often acting as both producer and judge.

Relevant current surfaces:

- `.claude/commands/test-pipeline.md`
- `.claude/commands/test-drive.md`
- `packages/build-cli/src/commands/assemble-pipeline.ts`
- `casts/claude/skills/pipeline-*/_assembly.json`
- per-skill `_provenance.json` and `_verify.json`

This means we exercise the words in cast skills, but do not black-box test installed skill activation, the packaged Pipeline harness, clean phase isolation, or independent evaluation.

## Proposed direction

Build a lightweight, Foundry-owned TypeScript controller backed by [Pi](https://pi.dev/) as the isolated agent worker runtime.

Pi should **not** decide which Pipeline phase runs next and should **not** grade its own output. The deterministic controller should own:

- Pipeline ordering from `_assembly.json`;
- one fresh worker process per phase or loop iteration;
- declared artifact handoffs;
- loop and branch predicates;
- time, tool, token, cost, and iteration budgets;
- deterministic validation;
- normalized run records and CI output.

Each Pi worker should receive:

- one explicitly loaded published Agent Skill;
- only the declared inputs for that phase;
- a fresh scratch directory;
- an explicit `/skill:` invocation;
- an allowlisted tool set and pinned provider/model configuration.

The runner should support an explicit isolation policy:

- `--sandbox local` (default) — a fresh process and scratch directory with Pi session, context-file, extension, prompt-template, and ambient skill discovery disabled. This is fast diagnostic isolation, not a security boundary.
- `--sandbox container` — run the worker in a disposable container with the selected skill bundle and declared inputs mounted read-only, one output directory mounted read-write, a fresh Pi configuration directory, and no checkout mount. Accept an optional `--sandbox-image ` and record the resolved image digest and mount manifest.

Container mode is the clean-room conformance option for initial tests and credential-bearing unattended runs; it need not be required for every local refinement run. Its credential and network policy must be explicit and recorded because Pi still needs provider access and some skills legitimately require remote services.

In container mode, workers should not be able to see authored Molds, `eval.md`, scenario expectations, `.git`, unrelated skills, or prior run artifacts. In local mode these resources should not be discovered or supplied, but their inaccessibility is not a security guarantee. Only declared, validated outputs should be promoted to the next phase.

Package the Pi-specific integration as a small Foundry-owned Pi package, tentatively `@galaxy-foundry/pi-harness`. Its extension should register a purpose-built `foundry_subagent` tool for the top-level Pipeline agent. One tool call runs one named published skill in a distinct ephemeral Pi child process (or one loop iteration), using the same worker runner and `local|container` sandbox backend as trace mode.

The extension is an execution adapter, not an orchestrator:

- the parent Pipeline agent chooses which skill to request and when;
- the extension resolves that name only against the installed Foundry skill package and rejects arbitrary skill paths;
- it validates path/mount boundaries, launches the child with exact skill loading, streams structured events, propagates cancellation and budgets, and returns a short structured result;
- it records the requested skill order, child runtime identity, usage, trace path, artifact hashes, and status;
- it does **not** select the next phase, repair ordering, retry a behavioral failure, evaluate output, or silently substitute another skill.

This preserves the black-box question: did the published top-level harness ask for the right nested skill in the right order? The extension makes that request real without answering it on the harness's behalf.

Evaluation should happen after execution:

1. deterministic schema/CLI checks run as ordinary processes;
2. qualitative properties go to a separate read-only judge session;
3. the producer transcript is retained for diagnostics but is not the judge's working context.

## Why Pi

Pi provides the missing execution seam without imposing a production orchestration architecture:

- Agent Skills support, including Claude/Codex skill directories;
- TypeScript extensions that can register tools and spawn subprocesses;
- exact skill loading with `--no-skills --skill `;
- explicit `/skill:name` activation;
- ephemeral `--no-session` runs;
- headless JSON/RPC events with tool calls, errors, and usage;
- provider/model/thinking selection;
- a TypeScript SDK if subprocess RPC becomes limiting.

References:

- [Skills](https://pi.dev/docs/latest/skills)
- [RPC mode](https://pi.dev/docs/latest/rpc)
- [JSON event stream](https://pi.dev/docs/latest/json)
- [SDK](https://pi.dev/docs/latest/sdk)
- [Security and containerization](https://pi.dev/docs/latest/containerization)

Pi is currently fast-moving and pre-1.0, so pin the exact package version and isolate it behind a small adapter. Pi also has no built-in filesystem/process/network sandbox; a temporary directory is useful but is not a security boundary. Credential-bearing unattended runs need a disposable container or equivalent policy sandbox.

## Two complementary modes

### Trace/conformance mode

The primary diagnostic mode walks `_assembly.json` and invokes every named published skill in a fresh Pi process. This guarantees per-phase isolation and localizes failures precisely.

### Black-box package mode

Install or unpack the actual Foundry plugin, load only its top-level `pipeline-*` skill plus the Foundry Pi extension, and invoke the Pipeline with `--use-subagents` as a consumer would. The parent agent sees the `foundry_subagent` tool; each call resolves and runs the requested packaged phase skill in a fresh child. This tests top-level dispatch, nested skill selection, run options, harness prose, and actual context isolation without pre-programming the expected phase order into the extension.

Neither mode replaces the other: trace mode proves phase behavior; black-box mode proves the packaged harness journey.

## Implementation sequence

### 1. Single-skill Pi runner

- Pin `@earendil-works/pi-coding-agent` exactly.
- Add a small RPC adapter under `@galaxy-foundry/build-cli`.
- Add a Foundry Pi package/extension, tentatively `@galaxy-foundry/pi-harness`, with a `foundry_subagent` tool backed by the same runner rather than a second subprocess implementation.
- Run one cast skill with no persistent session or ambient resource discovery (`--no-session --no-context-files --no-extensions --no-prompt-templates --no-skills --skill ` and a fresh `PI_CODING_AGENT_DIR`).
- Implement `--sandbox local|container`, defaulting to `local`; container mode mounts only the selected skill, declared inputs, and output directory.
- Capture raw JSONL, stderr, model/runtime identity, skill/provenance hashes, usage, timing, and artifact hashes.
- Record the sandbox mode and, for container runs, the image digest, mount manifest, network policy, and credential policy.
- Enforce a wall timeout and validate expected artifacts.
- Pilot with `summarize-nextflow` against one existing small fixture.

Success criterion: with `--sandbox container`, the run still works when `content/molds/`, `.git`, and `_emulated-runs/` are not mounted into the worker.

### 2. Linear Pipeline execution

Add a command shaped approximately like:

```sh
foundry-build test-pipeline nextflow-to-galaxy \
--scenario "nf-core/demo end to end" \
--engine pi \
--sandbox container \
--provider anthropic \
--model
```

- Walk `_assembly.json` in order.
- Launch one fresh worker per phase.
- Pass declared artifacts only.
- Add `--through ` for inexpensive slices and `--trials N` for repeated runs.
- Emit normalized `run.json` plus per-phase traces.
- Pilot `nextflow-to-galaxy` through its first two phases before attempting the full journey.

### 3. Executable control-flow metadata

Enrich assembly/runtime metadata only where necessary:

- phase inputs and required/optional outputs;
- nested skill dependencies for orchestrator skills;
- deterministic loop-completion predicate and maximum iterations;
- structured branch acceptance/fallback result;
- structured phase completion status.

The controller, not worker prose, must decide whether a loop or branch advances.

### 4. Independent grading

- Parse Pipeline/Mold eval properties and selected scenario expectations outside the workers.
- Run all available deterministic checks rather than awarding passes by inspection.
- Treat an unimplemented deterministic property as `unsupported`/`incomplete`, never as green.
- Use a separate read-only agent only for `llm-judged` properties.
- Emit `eval.json` and JUnit output, distinguishing infrastructure, skill, workflow, and grader failures.

### 5. Black-box and CI lanes

- Add a packaged-plugin black-box run under Pi that loads only the top-level Pipeline skill and the `foundry_subagent` extension.
- Assert from the trace that the parent requested the expected nested skills in order; do not have the extension enforce or repair that order.
- Keep the first black-box milestone linear. Add explicit loop-iteration and branch-chain semantics after the single-skill tool contract is stable.
- Retain small native Claude Code and Codex smoke tests because a Pi pass does not prove identical host behavior.
- Start with non-blocking trusted-branch smoke tests.
- Run representative multi-trial scenarios nightly.
- Reserve full Galaxy/Planemo journeys for opt-in/nightly/release lanes until cost and network reliability are understood.

## Acceptance criteria

- [ ] Every phase and loop iteration uses a distinct ephemeral worker session.
- [x] The Foundry Pi extension exposes one constrained subagent tool backed by the same worker runner as trace mode.
- [x] The extension resolves only installed Foundry skills and cannot accept arbitrary skill paths.
- [ ] In black-box mode, the parent agent—not the extension—selects the requested skill and order, and the trace records both.
- [x] `--sandbox local` disables ambient Pi resource discovery and is clearly reported as not being a security boundary.
- [x] `--sandbox container` uses a disposable container with no checkout mount; only the selected skill and declared inputs are mounted read-only, and only the output directory is writable.
- [x] Container workers cannot access prior runs, authored Molds, evals, scenarios, unrelated skills, or `.git`.
- [x] The sandbox mode and effective image, mounts, network, and credential policies are recorded in the run metadata.
- [x] The trace proves explicit activation of the expected published skill.
- [x] Only declared artifacts cross phase boundaries.
- [x] Artifacts are validated before downstream consumption.
- [ ] Loop completion and branch advancement use controller-owned predicates.
- [ ] Required deterministic checks actually execute.
- [ ] Producers do not grade themselves.
- [ ] Pi, model, CLI, fixture, cast, and artifact identities are recorded.
- [ ] Wall-clock, turn/tool, token/cost, loop, and output limits are enforced.
- [x] Raw traces and a versioned normalized run record are retained.
- [ ] Infrastructure errors, behavioral failures, and grader failures are distinct.
- [ ] Secret-bearing CI never executes untrusted pull-request code.
- [ ] Native Claude/Codex compatibility remains a separate smoke lane.

## Non-goals

- Selecting the production harness architecture.
- Turning Pipelines into Molds or moving orchestration into casting.
- Replacing exploratory `/test-drive` refinement runs.
- Requiring Pi for ordinary validation, casting, or contributor workflows.
- Requiring container isolation for every local run.

The first milestone should remain deliberately small: prove that an external clean-room CLI agent can explicitly activate one real cast skill, produce its declared artifact, and leave an auditable trace. Then extend the same controller to a two-phase Pipeline slice.

Start with .claude/commands/test-pipeline.md, .claude/commands/test-drive.md, packages/build-cli/src/commands/assemble-pipeline.ts, and the cast _assembly.json files. Run the proposed single-skill summarize-nextflow pilot first, then trace how linear execution, sandboxing, artifact validation, and independent evaluation should fit together. Done means the acceptance criteria are implemented and the container run works without checkout, prior-run, or unrelated-skill access.

Written by the indexing model from the issue text.