Agent Learning Track baseline run (GPT-5.1, OpenAI API judge+sim) — results + gap vs leaderboard

Summary

We ran the Agent Learning Track no-memory baseline using GPT-5.1 as the agent and GPT-5.4 for both the locked judge and user simulator, following the Agent Learning Track protocol. The only deviation from the documented setup is that we used the OpenAI API directly rather than Azure OpenAI for the judge and simulator client. Our results are ~6pp below the leaderboard's verified GPT-5.1 (no-memory) entry, and we wanted to share the data and ask whether the gap is expected.

Run Configuration

Baseline Results

Question

Is the ~6pp Pass@1 gap and ~0.4 UX gap expected when running the judge and simulator via the OpenAI API rather than Azure OpenAI? We want to confirm whether Azure vs OpenAI API routing for the locked eval client is a known source of variance before submitting our memory-augmented results.

We note our UX scores are now real per-dimension values (not defaulting to 3.0), so the judge is functioning correctly via OpenAI API. The gap appears to be consistent across all three domains rather than isolated to one, which makes us wonder if there is a systematic difference in judge/simulator behavior between Azure and OpenAI API endpoints for GPT-5.4.

Any guidance would be appreciated. Happy to share more details or trajectories if useful.

Start with the documented Agent Learning Track protocol and the locked gpt54.json configuration, then compare the OpenAI API and Azure OpenAI judge/simulator routing used for the run. Reproduce the baseline with the listed settings across customer_support, shopping_assistant, and travel, comparing Pass@1, Pass^5, and UX scores with the leaderboard values. Done means explaining whether endpoint routing accounts for the gap or identifying another variance.