how to online_mind2web eval protocol

Thanks for open-sourcing the evaluation framework.

I wanted to confirm the exact judge configuration behind the Online-Mind2Web numbers in the Fara1.5 report (Table 3). Section 4.1.1 states that you use the WebJudge of Xue et al. (2025) backed by o4-mini, and the vendored code under webeval/src/webeval/benchmarks/om2w/impl/src/ matches the official three-stage implementation. I want to rule out the AgentTrek protocol (src/methods/agenttrek_eval.py in the OSU repo), which some other work has used instead — as far as I can tell it is not used here, but confirmation would help.

Running Online-Mind2Web through web judge, I get ~33% with zero tasks blocked every trajectory completes and gets scored, so this isn't a site-access problem. That's far below the reported [63.4 for Fara1.5-9B / 33%].

Is there a setup step I'm missing for this path, or does it require a separate judge endpoint config?

Thanks.

Start with the Online-Mind2Web implementation under webeval/src/webeval/benchmarks/om2w/impl/src/ and compare it with src/methods/agenttrek_eval.py. Check Section 4.1.1 and Table 3 against the available judge and endpoint configuration. Done means confirming the protocol and documenting any required setup or separate judge configuration.