NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

feat: bound how long a routed request may take

Open
#277 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Summary

A routed request has no upper bound on how long it can take. There are two places a bound is needed, and this issue tracks both. Supersedes #346.

1. A total timeout on the upstream request

crates/switchyard-server/src/config.rs has no per-client timeout setting. An upstream that accepts a request and then responds slowly holds the connection for as long as it likes.

Relationship to #271. That PR adds a connection timeout and an idle-read timeout to TranslatingLlmClient. Both are needed and neither bounds total duration: an idle-read timeout resets on every successful read, so an upstream that emits one token every 100 seconds never trips it while keeping the request open indefinitely.

The proposal here is a total per-request bound, configurable per client:

[llm_clients.weak]
timeout_secs = 600

If #271 lands first this should be added to its HttpTransportConfig rather than as a separate mechanism. I have commented on #271 to that effect.

One consequence to note: since #316 moved token counting to an inherent method that still goes through send_encoded, a per-client timeout would also bound count_tokens.

2. A deadline on the judge consultation

For llm_classifier routes, the judge call runs before the routed call. JudgeClassifier::verdict in crates/libsy/src/algorithms/util/llm_judge.rs has no time limit, so a slow judge delays a request that the user is waiting on for content the judge does not produce.

Proposal: judge_deadline_ms on the route. On expiry the judge is treated as unavailable and the route falls back through the existing path used for any other judge failure, so no new behaviour is introduced. Unset by default.

Two details:

  • The bound must cover the whole consultation, not only the HTTP call. A judge that returns headers promptly and then stops mid-stream would otherwise hold the turn for the same length of time.
  • Expiry should be reported through the existing fail-open reporting (report_fail_open in the same file), so it appears in the switchyard.classifier_fail_open metric alongside the other reasons rather than as an untracked path.

judge_deadline_ms = 0 should be rejected at load time rather than treated as "no deadline".

Sequencing

These are separate changes and will be submitted as separate pull requests. The judge deadline has no external dependency. The client timeout waits on #271.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/switchyard-server/src/config.rs for per-client settings and crates/libsy/src/algorithms/util/llm_judge.rs, especially JudgeClassifier::verdict and report_fail_open. Trace the existing judge-failure fallback and streaming behavior before separating the two changes. Done means total client duration and whole-consultation judge deadlines are configurable, invalid zero deadlines are rejected, and expiry uses the existing fail-open metric path.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.