NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
feat(routing): add a TypeSafe-backed classifier router
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Summary
Explore and implement a TypeSafe System One routing option for Switchyard. TypeSafe would make a typed Choice over configured target labels, while Switchyard remains responsible for executing the selected target.
Motivation
The existing LLM-classifier path uses a generative model for the routing decision. A small live smoke comparison found that TypeSafe made the same 10/10 clear routing decisions with materially lower observed decision latency:
| Router | Correct | Mean | Median | Observed p95 |
|---|---|---|---|---|
TypeSafe jev-latest |
10/10 | 281 ms | 278 ms | 375 ms |
azure/openai/gpt-5.6-sol |
10/10 | 1,653 ms | 1,575 ms | 2,576 ms |
This is exploratory evidence, not a production-quality accuracy result. The cases were deliberately separable and each request ran once.
Proposed behavior
- Send normalized request state to TypeSafe as one typed
Choice. - Define each configured target with a stable label and semantic criterion.
- Route high-confidence choices to the selected target.
- Route low-confidence choices and TypeSafe service failures to a configured capable fallback.
- Load the TypeSafe credential from an environment variable; never place it in TOML or logs.
- Preserve the full returned probability distribution for telemetry and later threshold calibration.
Architecture question
switchyard-libsy is intentionally I/O-free, while TypeSafe is an external HTTP judgment service. The implementation should preserve that boundary. Likely options are a generic external-decision step served by the runner, or a runner-owned classifier provider that returns a routing decision to libsy. Direct HTTP calls from a libsy algorithm would violate the current architecture.
Acceptance criteria
- Agree on the I/O boundary and configuration shape.
- Add a TypeSafe-backed routing implementation with environment-based authentication.
- Validate target labels, fallback target, and confidence threshold.
- Fail open to the configured capable target on low confidence or service failure.
- Add mocked contract, configuration, fallback, and server integration tests.
- Record decision latency, selected label, confidence, and failure/fallback reason without logging secrets.
- Evaluate on representative labeled Switchyard traffic before making production quality claims.
- Document setup and operational behavior.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the switchyard-libsy I/O boundary and the runner integration points, then resolve whether the external decision belongs in a runner-owned provider or a generic external-decision step. Define the configuration and fallback behavior before implementation. Done means mocked contract, configuration, fallback, and server integration tests pass, telemetry and secret handling are covered, and setup and operational behavior are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100