Discrepancy of gemini-3.5-flash public-set pass rate: independent rerun measures 29.9% vs README's 14.83%

Start with scripts/eval.py around lines 196–207 and trace the Gemini path into clients.py. Check which proxy endpoint and sampling branch the README run used, then inspect per-rollout errors, aborts, effective thinking level, rollout count, and run timing. Compare the available per-task results to determine what explains the 14.83% versus 29.92% discrepancy.

Hi team!
Thanks for sharing the public scores in README. We independently reran the public 600-task set to calibrate against the pass rates recently published in the README. Six of the seven models we share with your table reproduce within 0.04–6.8pp — but gemini-3.5-flash comes out at 29.9% on our side vs 14.83% in the README, a 2× gap that stands completely alone.

Our setup

• Task content: commit id = eda21410
• Scoring: your own code, vendored byte-identically — the reported number is the mean of your task_completed_correctly over the 600 scored tasks.
• Config: api toolset (api_search / api_fetch / base64_encode), 50-step cap, temperature 1.0, 4 rollouts per task (2,400 graded rollouts per model).
• Gemini is called through the native google.genai SDK: thinkingLevel=HIGH, 65,536 max output tokens, and thought signatures re-attached across turns.

Results on shared models (your README pass rate vs ours)

Your own private leaderboard corroborates our number

Comparing the README's public-set pass rates to the private-set "Score" on zapier.com/benchmarks, every model completes roughly half as many private tasks as public ones — except gemini-3.5-flash, whose private score (14.5%) is nearly equal to its public one (14.83%):

Against the README's public number, gemini is a wild outlier (0.98 vs the 0.5–0.7 band). Against our public number, its ratio (0.48) falls exactly in line with every other model. In other words: your own private-leaderboard score for flash predicts a public-set pass rate of ~29–30%, which is what we measure.

Where Claude Code suspect the divergence

The runner has no native Google client — Gemini goes through OpenAI chat-completions to a proxy. Two flash-specific hazards in scripts/eval.py:

1. The thinkingLevel special case is keyed on the literal substring "litellm" in base_url (eval.py ~L199–207). If the gateway URL doesn't contain that substring, flash models fall through to sampling_args={"reasoning_effort": ...} — which the comment directly above (~L196) notes LiteLLM rejects for vertex gemini flash variants. Depending on the proxy that either 400s rollouts (non-retryable in clients.py) or silently yields a different effective thinking level than "high".
2. Thought-signature / reasoning continuity is not round-tripped on the OpenAI-compat path. Gemini relies on thought signatures for multi-turn tool planning; losing them degrades exactly the long-horizon behavior this benchmark measures, while leaving non-Gemini models untouched.

Questions

1. Which endpoint/proxy did the README's flash run use — and does its base_url contain "litellm" (i.e., which branch of the _gemini_flash_needs_native check did it take)?
2. Could you check the flash run's per-rollout error/abort rate and the effective thinkingLevel that reached the API?
3. How many rollouts per task back the README numbers, and roughly when did the runs execute (model snapshot drift)?

Happy to share our per-task pass-rate vector (600 task ids × pass rate) so we can diff exactly where the two harnesses disagree — the divergence should be concentrated and diagnosable. Thanks for publishing the public scores; the calibration exercise has been genuinely useful, and flash looks a lot stronger on this benchmark than the current table suggests.