NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

feat(libsy): evict targets that reject a request as unservable

Open
#345 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Problem

The fallback ladder in call_model_with_fallback recognises two reasons a target can fail a route:

RoutingFallbackReason::ContextWindow => evictions.record(identity, failed),  // session-sticky
RoutingFallbackReason::Unavailable   => target_unavailable(&request, failed), // request-local

There is a third case it does not model: a target that rejects the request as one it cannot serve at all — an unsupported modality, a tool schema it does not accept, a feature the model lacks. Providers return these as a 400, so today they fall through classify_fallback's _ => return None and surface to the caller as a hard error. The route has another target that could have served the turn and never tries it.

Proposal

A CapabilityRejected { model, message } variant on LlmClientError, detected at the same place ContextWindowExceeded is (a 400 whose body matches a backend-supplied predicate), and a third RoutingFallbackReason::Capability.

The routing question is which of the two existing behaviours it should share, and the answer is not the obvious one. A capability reject looks like an availability problem — the target refused the request — but it is a permanent property of that target for this conversation. Unavailable is request-local, so mapping it there re-probes the rejecting target on every subsequent turn, paying a guaranteed failure per turn for the rest of the session. It belongs with ContextWindow:

RoutingFallbackReason::ContextWindow | RoutingFallbackReason::Capability => {
    evictions.record(identity, failed)
}

Both are "this target cannot serve this conversation", which is exactly what SessionEvictions records.

This distinction compiles either way and is invisible in a single-turn test, so it is worth pinning with a test that asserts the rejecting target is probed exactly once across three turns. I have verified that test fails if the variant is routed to Unavailable instead.

Scope

  • crates/protocol/src/client.rs — new LlmClientError variant and RoutingFallbackReason::Capability (as_str()"capability"). This is a public-API addition, which is the main thing worth a design opinion before a PR.
  • crates/libsy-llm-client/src/{backend,client}.rs — detection, ordered after the overflow check because an oversized request is recoverable on the same target and a capability reject never is.
  • crates/libsy/src/core/algorithm.rsclassify_fallback mapping and the dispatch above.
  • crates/switchyard-server/src/lib.rs — error mapping.

Related: #298 introduced the reason-based ladder this extends; #273 touches the same fall_through.rs region.

Implementation is rebased onto current main and passing fmt/clippy -D warnings/test --workspace (including --test-threads=1). Raising it as an issue first rather than opening the PR, since the protocol addition deserves a design call.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/protocol/src/client.rs and the ContextWindowExceeded detection in crates/libsy-llm-client/src/{backend,client}.rs, then trace classify_fallback and call_model_with_fallback in crates/libsy/src/core/algorithm.rs. Check the error mapping in crates/switchyard-server/src/lib.rs. Done means capability rejections follow the intended session eviction behavior, with a test proving the target is probed once across three turns, plus the stated workspace checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.