Runner trial: pi vs codex vs haiku on the same task batch

The experiment phase of #8, now that #10 is done. Blocked on #11 (the ticker cannot show pi activity yet) and #12 (no quota measurement without per-task usage capture). #13 is not a blocker.

Since both pi and codex bill to the same ChatGPT subscription, the question is no longer whether to offload quota from the Claude plan (codex already does) but which harness earns the traffic on that pool, with haiku as the same-pool Claude baseline.

Phase 1: swarm A/B

• Pick four to six real, mechanical, well-specified issues (config changes, small functions, docs), the class of work a cheap runner should own
• Run the batch through each runner sequentially: thrawn swarm <issues> --runner codex, review, thrawn abort swarm, then the same batch with --runner pi (sequential because both swarms would claim the same thrawn/swarm/<issue> branch names)
• Score each branch: adopt rate (mergeable as-is), checks pass rate, retry or intervention rate, quota burn per shipped task and wall clock

Phase 2: planner routing

• Put pi and codex in the planner's rotation via their notes and route mechanical tasks for a fortnight
• Let the ledger accumulate the same metrics on real work, including whether runs leave the account rate limited

Decision

• Decide from the numbers and record the outcome on #8: which runner gets mechanical traffic, or a split (for example codex for autonomy, pi for instrumentation)

What better looks like

pi does not need to write better code than codex. pi wins if it ships comparable work while burning less of the shared window (smaller system prompt) or if its structured usage data and session resume prove their worth. codex wins if its tuned harness needs fewer interventions. haiku wins if the Claude pool is roomy enough that offloading was never worth the complexity.

🤖 Generated with Claude Code

Start by reading #8 and the status of blockers #11 and #12, then inspect the planner and ledger entry points mentioned in the issue. Run the specified sequential thrawn swarm <issues> --runner codex and --runner pi batches only when those blockers are resolved. Done means the fortnight metrics are recorded and the runner decision is documented on #8.