OrcaRouter provider support for MolmoWeb's GPT/OpenAI-compatible agents and judges
- Dominant language
- Python
- Stars
- 591
- Forks
- 81
- Avg merge
- 9d 22h
- Merged PRs (30d)
- 1
Description
## Proposal: OrcaRouter as an optional provider for GPT/OpenAI-compatible agents and judges
MolmoWeb gives researchers something most web-agent repos don't: a reproducible, two-stage
evaluation pipeline where the same codebase runs any supported agent across WebVoyager,
Online Mind2Web, Odysseys, DeepShop, WebTailBench, or custom tasks, then scores the
trajectories with an LLM judge. The multi-provider design — local MolmoWeb checkpoints
alongside `gemini_cua`, `gemini_axtree`, and `gpt_axtree` agents gated by standard
`GOOGLE_API_KEY` / `OPENAI_API_KEY` env vars — is what lets you compare a locally served
8B model against frontier API agents on identical tasks, exactly the kind of abstraction
that keeps model choices easy to swap.
That same abstraction is why I'd like to propose [OrcaRouter](https://www.orcarouter.ai)
as an optional additional provider. For teams running the GPT-based agents or the
OpenAI-backed judges, an extra OpenAI-compatible endpoint lets them compare GPT-5 against
other chat, reasoning, and image models without touching the MolmoWeb local inference
path. This is a pure addition: no existing provider, env var, or default model would change.
### What is proposed
- Add OrcaRouter as an optional provider for the API-based agents and OpenAI-compatible
judges, following how the repo already configures them today. For example,
`agent/gpt_axtree_agent.py` builds a module-level `OpenAI()` client and reads a model
from `GPT_AXTREE_MODEL`; `benchmarks/judges/webvoyager_judge.py` does the same with the
OpenAI SDK (`openai>=2.8.1` is already a dependency in `pyproject.toml`). Supporting
OrcaRouter there would be a small, opt-in change: point the client at OrcaRouter's
OpenAI-compatible base URL and use a standard API key.
- Concretely, the features most relevant to MolmoWeb users are:
- **One endpoint, many models**: chat, reasoning, and image models via one
OpenAI-compatible API, letting runs and judges switch model families without code
changes.
- **Automatic model routing and provider failover**, so a multi-day benchmark run can
survive a provider outage rather than fail mid-trajectory.
- **Usage tracking and budgets**, useful when fanning out `num_workers` judges/trajectory
runs across a team or lab.
- I'm an engineer on the OrcaRouter team, and I'm raising this as a request for maintainer
input before any implementation. Nothing here is implemented or tested in this repo —
this is a proposal only.
### Disclosure
OrcaRouter runs an optional open-source partner program under which approved OSS projects
can receive a 5% revenue share from OrcaRouter usage attributed to their integration. To
be clear: participation is not a prerequisite for integration, and I'd be happy to follow
whatever disclosure or governance requirements the MolmoWeb/Ai2 maintainers prefer.
OrcaRouter is already used by a number of open-source projects such as RAGFlow, Dify,
goose, and NocoBase; see https://www.orcarouter.ai/built-with for the current list.
I'd welcome your thoughts on whether this direction fits MolmoWeb's roadmap, and if so I'd
be glad to submit an implementation PR for review.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.