Question about STATE-Bench submission, external models, and correction retries

Hello STATE-Bench team,

We are developing Capybara, a local system for building and evaluating tool-using AI agents, and are evaluating it locally with the STATE-Bench Agent Learning Track.

The Capybara STATE-Bench integration is planned to be published at:

https://github.com/pkc000pkc/capybara/tree/main/examples/state-bench

The integration includes:

• An automatic runtime memory hook before every agent loop;
• A training hook for extracting reusable experiences;
• A failure-correction hook;
• A replay-based admission hook;
• Cross-session variables and immutable snapshots;
• An append-only JSONL token ledger for all model roles.

The correction process has the following constraints:

• The correction hook receives only observable conversation, tool calls, and sanitized failure status;
• It does not receive reference answers, target states, Judge explanations, expected tools, or hidden evaluator data;
• Every correction attempt starts from a fresh clean environment;
• The number of correction attempts is limited to 3, 4, or 5;
• Experiences are not promoted to long-term memory unless a closed-book replay passes.

Runtime configuration:

• STATE-Bench: 0.8.1
• Benchmark commit: 5644b1838d96bc4483da29642d058ecaa6f80f7f
• Agent model: gpt-5.6-luna
• Simulator/Judge: gpt-5.4
• Client: OpenAI-compatible Responses API client
• Training backend: http://127.0.0.1:3005
• Memory prefetch: up to 2 entries, 450 estimated tokens per loop, 1,200 per run
• Token roles: agent, simulator, task_judge, ux_judge, correction_hook

Current results are local, non-official diagnostic results:

• 15-task diagnostic: 46.7% baseline pass@1 and 53.3% frozen-learning pass@1
• 3-task memory smoke: 33.3% baseline pass@1 and 66.7% frozen-learning pass@1
• pass^5 was not measured in these runs
• The runs used an OpenAI-compatible proxy instead of the official Azure evaluation client
• The planned full run is 50 tasks per domain, 5 runs per task, across 3 domains, for 750 trajectories

We would like to clarify:

1. Can an external model be used as the agent while the official GPT-5.4 simulator and Judge remain locked?
2. Is an OpenAI-compatible client acceptable for an official submission?
3. Is a correction process based only on observable trajectory evidence and a generic failure signal allowed?
4. How should 3-5 correction retries be counted for pass@1 and pass^5?
5. Which trajectories, metrics, token reports, and run metadata are required?
6. Is there a private channel for submitting results and evaluation artifacts?

We will not post API keys, evaluation credentials, private endpoints, or complete task trajectories in this issue. We can provide sanitized reports and a code snapshot if needed.

Thank you.

Start with the STATE-Bench 0.8.1 documentation and the submission guidance associated with benchmark commit 5644b1838d96bc4483da29642d058ecaa6f80f7f. Review the planned examples/state-bench integration and the official Azure evaluation client requirements. Done means the external-model, client, correction-retry, metric, artifact, and submission-channel questions have documented answers.