allenai / allenai/molmoweb

OrcaRouter provider support for MolmoWeb's GPT/OpenAI-compatible agents and judges

Aberta
#27 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
592
Forks
81
Merge médio
9d 22h
PRs com merge (30d)
1

Descrição

## Proposal: OrcaRouter as an optional provider for GPT/OpenAI-compatible agents and judges

MolmoWeb gives researchers something most web-agent repos don't: a reproducible, two-stage
evaluation pipeline where the same codebase runs any supported agent across WebVoyager,
Online Mind2Web, Odysseys, DeepShop, WebTailBench, or custom tasks, then scores the
trajectories with an LLM judge. The multi-provider design — local MolmoWeb checkpoints
alongside `gemini_cua`, `gemini_axtree`, and `gpt_axtree` agents gated by standard
`GOOGLE_API_KEY` / `OPENAI_API_KEY` env vars — is what lets you compare a locally served
8B model against frontier API agents on identical tasks, exactly the kind of abstraction
that keeps model choices easy to swap.

That same abstraction is why I'd like to propose [OrcaRouter](https://www.orcarouter.ai)
as an optional additional provider. For teams running the GPT-based agents or the
OpenAI-backed judges, an extra OpenAI-compatible endpoint lets them compare GPT-5 against
other chat, reasoning, and image models without touching the MolmoWeb local inference
path. This is a pure addition: no existing provider, env var, or default model would change.

### What is proposed

- Add OrcaRouter as an optional provider for the API-based agents and OpenAI-compatible
judges, following how the repo already configures them today. For example,
`agent/gpt_axtree_agent.py` builds a module-level `OpenAI()` client and reads a model
from `GPT_AXTREE_MODEL`; `benchmarks/judges/webvoyager_judge.py` does the same with the
OpenAI SDK (`openai>=2.8.1` is already a dependency in `pyproject.toml`). Supporting
OrcaRouter there would be a small, opt-in change: point the client at OrcaRouter's
OpenAI-compatible base URL and use a standard API key.
- Concretely, the features most relevant to MolmoWeb users are:
- **One endpoint, many models**: chat, reasoning, and image models via one
OpenAI-compatible API, letting runs and judges switch model families without code
changes.
- **Automatic model routing and provider failover**, so a multi-day benchmark run can
survive a provider outage rather than fail mid-trajectory.
- **Usage tracking and budgets**, useful when fanning out `num_workers` judges/trajectory
runs across a team or lab.
- I'm an engineer on the OrcaRouter team, and I'm raising this as a request for maintainer
input before any implementation. Nothing here is implemented or tested in this repo —
this is a proposal only.

### Disclosure

OrcaRouter runs an optional open-source partner program under which approved OSS projects
can receive a 5% revenue share from OrcaRouter usage attributed to their integration. To
be clear: participation is not a prerequisite for integration, and I'd be happy to follow
whatever disclosure or governance requirements the MolmoWeb/Ai2 maintainers prefer.
OrcaRouter is already used by a number of open-source projects such as RAGFlow, Dify,
goose, and NocoBase; see https://www.orcarouter.ai/built-with for the current list.

I'd welcome your thoughts on whether this direction fits MolmoWeb's roadmap, and if so I'd
be glad to submit an implementation PR for review.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Start by reading agent/gpt_axtree_agent.py, benchmarks/judges/webvoyager_judge.py, and the OpenAI dependency in pyproject.toml to understand the existing provider configuration. Confirm the maintainers’ desired scope for OrcaRouter, including endpoint, key, and model handling; done means an approved optional integration that preserves existing providers, environment variables, and default models.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
python
Domínio
api, backend
Tipo de issue
Funcionalidade
Dificuldade
5/5
Tempo estimado
Mais de uma semana
Status de atividade
Ativa
Clareza
Razoavelmente clara
Facilidade para iniciantes
35/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.