NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[feature] Stackable / Hierarchical routers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Problem
For cost efficient but low-latency routing, using staged_router as a deterministic step for weak/strong is useful, but having the strong target as the llm classifer would allow a greater granularity of cost control and potentially of PII management. If the deterministic step indicates esclation to the llm classifer, then upstream classification can be configured to control cloud costs.
Proposed solution
This would be simply that the "strong" target can route to the llm classifier natively, without the need for an intermediate step and would give native visibility of routing decisions in both the deterministic and probabilstic methods, giving users a single pane of a glass to configure costs and cost savings.
Additionally, a mechanism to optimise prompts being sent to the llm_classifer could be of value. Going further, having the llm classifer given a recipie for adaptive training to override the staged_router heuristics and potentially return back to the staged router weak target/local, with an explicit instruction to process locally. Thbis could be used in the case where the llm classifer determines there could be PII in the prompt, as a secondary control for users who have DLP/PII deterministic classifiers between the staged_router and the llm classifer, for additional control against sensitive information leakage upsteam to cloud.
Alternatives considered
I looked at using staged router to use litellm as the capable/strong target, but visibility of routing decisions is fragmented.
Scope notes
- Which surface owns the change: server API or configuration, routing algorithm,
protocol type, translation codec, upstream client, skill-distillation contract,
Python binding, or launcher? - Does it change a public Rust, PyO3, Python, CLI, or deployment-TOML interface?
- Backward-compatibility concerns?
Additional context
Links to related issues, prior art in other routers, references.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the staged_router and llm_classifier implementations and their configuration or server API surfaces. Define the supported hierarchical routing behavior, visibility of routing decisions, cost and PII controls, and backward-compatibility expectations before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100