NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

feat(routing): add a TypeSafe-backed classifier router

Open
#723 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Summary

Explore and implement a TypeSafe System One routing option for Switchyard. TypeSafe would make a typed Choice over configured target labels, while Switchyard remains responsible for executing the selected target.

Motivation

The existing LLM-classifier path uses a generative model for the routing decision. A small live smoke comparison found that TypeSafe made the same 10/10 clear routing decisions with materially lower observed decision latency:

Router Correct Mean Median Observed p95
TypeSafe jev-latest 10/10 281 ms 278 ms 375 ms
azure/openai/gpt-5.6-sol 10/10 1,653 ms 1,575 ms 2,576 ms

This is exploratory evidence, not a production-quality accuracy result. The cases were deliberately separable and each request ran once.

Proposed behavior

  • Send normalized request state to TypeSafe as one typed Choice.
  • Define each configured target with a stable label and semantic criterion.
  • Route high-confidence choices to the selected target.
  • Route low-confidence choices and TypeSafe service failures to a configured capable fallback.
  • Load the TypeSafe credential from an environment variable; never place it in TOML or logs.
  • Preserve the full returned probability distribution for telemetry and later threshold calibration.

Architecture question

switchyard-libsy is intentionally I/O-free, while TypeSafe is an external HTTP judgment service. The implementation should preserve that boundary. Likely options are a generic external-decision step served by the runner, or a runner-owned classifier provider that returns a routing decision to libsy. Direct HTTP calls from a libsy algorithm would violate the current architecture.

Acceptance criteria

  • Agree on the I/O boundary and configuration shape.
  • Add a TypeSafe-backed routing implementation with environment-based authentication.
  • Validate target labels, fallback target, and confidence threshold.
  • Fail open to the configured capable target on low confidence or service failure.
  • Add mocked contract, configuration, fallback, and server integration tests.
  • Record decision latency, selected label, confidence, and failure/fallback reason without logging secrets.
  • Evaluate on representative labeled Switchyard traffic before making production quality claims.
  • Document setup and operational behavior.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the switchyard-libsy I/O boundary and the runner integration points, then resolve whether the external decision belongs in a runner-owned provider or a generic external-decision step. Define the configuration and fallback behavior before implementation. Done means mocked contract, configuration, fallback, and server integration tests pass, telemetry and secret handling are covered, and setup and operational behavior are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.