OrcaRouter provider support for MolmoWeb's GPT/OpenAI-compatible agents and judges
- Linguagem predominante
- Python
- Estrelas
- 592
- Forks
- 81
- Merge médio
- 9d 22h
- PRs com merge (30d)
- 1
Descrição
## Proposal: OrcaRouter as an optional provider for GPT/OpenAI-compatible agents and judges
MolmoWeb gives researchers something most web-agent repos don't: a reproducible, two-stage
evaluation pipeline where the same codebase runs any supported agent across WebVoyager,
Online Mind2Web, Odysseys, DeepShop, WebTailBench, or custom tasks, then scores the
trajectories with an LLM judge. The multi-provider design — local MolmoWeb checkpoints
alongside `gemini_cua`, `gemini_axtree`, and `gpt_axtree` agents gated by standard
`GOOGLE_API_KEY` / `OPENAI_API_KEY` env vars — is what lets you compare a locally served
8B model against frontier API agents on identical tasks, exactly the kind of abstraction
that keeps model choices easy to swap.
That same abstraction is why I'd like to propose [OrcaRouter](https://www.orcarouter.ai)
as an optional additional provider. For teams running the GPT-based agents or the
OpenAI-backed judges, an extra OpenAI-compatible endpoint lets them compare GPT-5 against
other chat, reasoning, and image models without touching the MolmoWeb local inference
path. This is a pure addition: no existing provider, env var, or default model would change.
### What is proposed
- Add OrcaRouter as an optional provider for the API-based agents and OpenAI-compatible
judges, following how the repo already configures them today. For example,
`agent/gpt_axtree_agent.py` builds a module-level `OpenAI()` client and reads a model
from `GPT_AXTREE_MODEL`; `benchmarks/judges/webvoyager_judge.py` does the same with the
OpenAI SDK (`openai>=2.8.1` is already a dependency in `pyproject.toml`). Supporting
OrcaRouter there would be a small, opt-in change: point the client at OrcaRouter's
OpenAI-compatible base URL and use a standard API key.
- Concretely, the features most relevant to MolmoWeb users are:
- **One endpoint, many models**: chat, reasoning, and image models via one
OpenAI-compatible API, letting runs and judges switch model families without code
changes.
- **Automatic model routing and provider failover**, so a multi-day benchmark run can
survive a provider outage rather than fail mid-trajectory.
- **Usage tracking and budgets**, useful when fanning out `num_workers` judges/trajectory
runs across a team or lab.
- I'm an engineer on the OrcaRouter team, and I'm raising this as a request for maintainer
input before any implementation. Nothing here is implemented or tested in this repo —
this is a proposal only.
### Disclosure
OrcaRouter runs an optional open-source partner program under which approved OSS projects
can receive a 5% revenue share from OrcaRouter usage attributed to their integration. To
be clear: participation is not a prerequisite for integration, and I'd be happy to follow
whatever disclosure or governance requirements the MolmoWeb/Ai2 maintainers prefer.
OrcaRouter is already used by a number of open-source projects such as RAGFlow, Dify,
goose, and NocoBase; see https://www.orcarouter.ai/built-with for the current list.
I'd welcome your thoughts on whether this direction fits MolmoWeb's roadmap, and if so I'd
be glad to submit an implementation PR for review.
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Direção de pesquisa
Start by reading agent/gpt_axtree_agent.py, benchmarks/judges/webvoyager_judge.py, and the OpenAI dependency in pyproject.toml to understand the existing provider configuration. Confirm the maintainers’ desired scope for OrcaRouter, including endpoint, key, and model handling; done means an approved optional integration that preserves existing providers, environment variables, and default models.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- python
- Domínio
- api, backend
- Tipo de issue
- Funcionalidade
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Status de atividade
- Ativa
- Clareza
- Razoavelmente clara
- Facilidade para iniciantes
- 35/100