NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
feat(libsy): evict targets that reject a request as unservable
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Problem
The fallback ladder in call_model_with_fallback recognises two reasons a target can fail a route:
RoutingFallbackReason::ContextWindow => evictions.record(identity, failed), // session-sticky
RoutingFallbackReason::Unavailable => target_unavailable(&request, failed), // request-local
There is a third case it does not model: a target that rejects the request as one it cannot serve at all — an unsupported modality, a tool schema it does not accept, a feature the model lacks. Providers return these as a 400, so today they fall through classify_fallback's _ => return None and surface to the caller as a hard error. The route has another target that could have served the turn and never tries it.
Proposal
A CapabilityRejected { model, message } variant on LlmClientError, detected at the same place ContextWindowExceeded is (a 400 whose body matches a backend-supplied predicate), and a third RoutingFallbackReason::Capability.
The routing question is which of the two existing behaviours it should share, and the answer is not the obvious one. A capability reject looks like an availability problem — the target refused the request — but it is a permanent property of that target for this conversation. Unavailable is request-local, so mapping it there re-probes the rejecting target on every subsequent turn, paying a guaranteed failure per turn for the rest of the session. It belongs with ContextWindow:
RoutingFallbackReason::ContextWindow | RoutingFallbackReason::Capability => {
evictions.record(identity, failed)
}
Both are "this target cannot serve this conversation", which is exactly what SessionEvictions records.
This distinction compiles either way and is invisible in a single-turn test, so it is worth pinning with a test that asserts the rejecting target is probed exactly once across three turns. I have verified that test fails if the variant is routed to Unavailable instead.
Scope
crates/protocol/src/client.rs— newLlmClientErrorvariant andRoutingFallbackReason::Capability(as_str()→"capability"). This is a public-API addition, which is the main thing worth a design opinion before a PR.crates/libsy-llm-client/src/{backend,client}.rs— detection, ordered after the overflow check because an oversized request is recoverable on the same target and a capability reject never is.crates/libsy/src/core/algorithm.rs—classify_fallbackmapping and the dispatch above.crates/switchyard-server/src/lib.rs— error mapping.
Related: #298 introduced the reason-based ladder this extends; #273 touches the same fall_through.rs region.
Implementation is rebased onto current main and passing fmt/clippy -D warnings/test --workspace (including --test-threads=1). Raising it as an issue first rather than opening the PR, since the protocol addition deserves a design call.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crates/protocol/src/client.rs and the ContextWindowExceeded detection in crates/libsy-llm-client/src/{backend,client}.rs, then trace classify_fallback and call_model_with_fallback in crates/libsy/src/core/algorithm.rs. Check the error mapping in crates/switchyard-server/src/lib.rs. Done means capability rejections follow the intended session eviction behavior, with a test proving the target is probed once across three turns, plus the stated workspace checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100