Feature proposal: reproducible task experiment mode for Codex

What variant of Codex are you using?

CLI / app-server

What feature would you like to see?

I would like a first-class experiment mode for running the same repository task against multiple Codex configurations from an identical starting point and comparing the outcomes.

Conceptually:

codex experiment task.md \
  --variant model=gpt-5.6-sol,reasoning=high \
  --variant model=gpt-5.6-terra,reasoning=high \
  --verify "pytest -q"

Each candidate would run in an isolated worktree from the same pinned Git commit, receive the same task/instructions, and differ only in the explicitly declared configuration dimensions.

A deliberately bounded first version could:

• pin the repository base SHA and create an isolated worktree per candidate;
• run the same prompt/instructions for every candidate;
• support a small configuration matrix such as model, reasoning effort, and collaboration mode;
• optionally run one or more user-supplied verification commands after each candidate finishes;
• never merge or modify the user's active worktree automatically;
• produce a human summary plus a versioned machine-readable result containing completion status, changed files/diffstat, verification results, duration, and usage/tool-cycle metadata where available;
• let the user inspect and choose a candidate, or discard all of them.

Why this would be useful

Today, comparing two Codex configurations on a real repository task requires external harness scripts and careful manual worktree/reset management. That makes controlled comparisons hard to reproduce and easy to contaminate with different starting state.

This would make local evaluation a normal Codex workflow: same task, same repository state, declared variables, isolated execution, explicit verification, comparable results.

This is different from proposals for parallel implementation/subagents whose purpose is to solve one task collaboratively. The purpose here is controlled experimentation and comparison; candidates remain isolated and Codex does not automatically combine them.

It could be useful for users choosing models/configurations, maintainers reproducing behavioural regressions, and teams building their own task-level evaluation sets without requiring a separate agent harness.

Scope / non-goals

• Not a global model benchmark or leaderboard.
• Not automatic selection based only on LLM judgement.
• Not automatic merging of a "winner".
• Verification remains user-defined and repository-local.
• Reports should avoid embedding secrets or raw prompt/tool content by default.

I searched current issues for task experiment/evaluation modes, comparative runs, multiple-model task execution, candidate implementations, and worktree-based comparisons. There are related requests around parallel implementations and benchmarking, but I did not find an equivalent proposal for controlled same-task configuration experiments.

Since this repository currently treats external code contributions as invitation-only, this is a product/design proposal rather than an implementation PR.

No source files, tests, or implementation entry points are identified in the proposal. Start by locating the CLI command dispatch and existing worktree or task-execution flows; done would require an agreed design and implementation for isolated variant runs, verification, and reproducible machine-readable results.