NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[feature] Serve intermediate judge calls in the LiteLLM routing plugin (classifier-backed routing)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
The litellm example plugin rejects any Step.CallModel, and its README rules out classifier-backed routers on that basis. That keeps llm_task_classifier out of the LiteLLM integration entirely.
We built a routing plugin that serves those intermediate calls instead of rejecting them: it drives run_stream(), answers each judge call in-plugin, narrows candidate_models from the RoutingOutcome, and reports judge errors with ModelCall.fail() so the classifier's own fallback kicks in. It works today against main (f4152d07) and litellm 1.100.0. In our deep-research workload, routine steps go to the efficient deployment (judge p_solve 0.90+) and a hard synthesis step escalates (p_solve 0.05–0.18). Cost is real, though: the judge adds 6–21s and ~1.4k prompt tokens per routed call.
I would be curious for the team's thoughts on a potential PR for this (we could upstream the changes we did for our plugin internally) or whether this is an intentional limitation. One design question: our judge uses its own OpenAI client, which bypasses LiteLLM's spend tracking and needs separate credentials. Routing the judge through LiteLLM against a designated non-routed deployment seems cleaner (recursion-safe, spend-tracked, one credential model), but an injectable judge client is also an option. Which shape would you want?
Happy to contribute the PR with tests if there's interest in this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the LiteLLM example plugin and its README, then trace run_stream() handling for Step.CallModel, RoutingOutcome, and ModelCall.fail(). Clarify whether the judge should use LiteLLM or an injectable client, and confirm the recursion, credential, and spend-tracking expectations. Done means the team agrees on the design and the behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100