NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

fix(libsy): cap the windowed judge payload and restate the routing instruction last

Open
#279 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Summary

When an llm_classifier route sets recent_turn_window, the judge is sent recent conversation in addition to the routing task. Two problems follow from how that payload is built, both in crates/libsy/src/algorithms/llm_class.rs.

Supersedes #349.

Problem 1: the payload has no size limit

trim_messages (llm_class.rs:84) selects the messages the judge sees and returns them unchanged. The window is counted in turns, and turn size varies widely: a window of 4 turns is a few hundred characters of conversation, or tens of thousands if one turn carries a large tool result. Judge cost and latency vary accordingly for a single fixed configuration, and a large tool result can crowd out the request being judged.

Switchyard already solves this for its other judge. crates/libsy/src/algorithms/util/escalation.rs, which builds the payload for the escalation router, has:

  • MAX_REQUEST_CHARS = 18_000, a cap on the whole payload
  • SYSTEM_CHARS = 1_000 and FIRST_USER_CHARS = 2_000, per-message caps on the parts worth protecting
  • truncate_middle and TRUNCATION_SUFFIX = "...<truncated>", so clipping is visible to the model
  • a loop in summarize_for_judge (escalation.rs:293) that drops the oldest window entry until the payload fits

The capability and custom classifier paths have none of this. The same kind of window is bounded for one judge and unbounded for another.

Problem 2: the routing instruction ends up buried

The judge is asked to route the user's request and answer with a routing JSON object. With a window set, that instruction is followed by assistant turns and tool output, and the judge tends to respond about the trailing content instead.

Example: a request of "check my account balance" followed by a window containing a long stack trace from a failed tool call. The verdict describes the stack trace rather than the balance lookup.

Restating the instruction as the last message fixes it. This is prompt ordering only; the text is a single sentence:

Route the conversation above. Output ONLY the routing JSON object, nothing else.

(An earlier revision of this issue called that "reinforcement", which invites confusion with reinforcement learning. It is unrelated.)

Proposal

Two changes, to be submitted as separate pull requests:

  1. Restate the routing instruction after windowed content. Only on the windowed path: with recent_turn_window unset, task_messages (llm_class.rs:150) sends the opening task and the latest user follow-up and no assistant or tool content, so there is nothing to crowd it out. The test should assert the instruction is the last message, not merely present, since position is the whole point.

  2. Cap the payload, reusing the shape of summarize_for_judge, and expose MAX_REQUEST_CHARS as a route-level judge_char_budget. Default it to the current value so no existing deployment changes behaviour.

Relationship to #487

#487 made classify_trigger = "user_turn" the natural setting for interactive agents, which is where a window is most likely to be configured. Both profiles added in benchmark/routing-profiles/ set recent_turn_window = 6.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with trim_messages and task_messages in crates/libsy/src/algorithms/llm_class.rs, then compare their payload handling with summarize_for_judge in crates/libsy/src/algorithms/util/escalation.rs. Done means the windowed instruction is the final message, the judge payload respects a route-level budget with visible truncation, and tests verify both behaviors.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.