NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[Feature] Separate the user's question from the context Claude Code injects around it
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Problem
When Switchyard runs behind Claude Code, the router often can't tell which
question it is supposed to route.
Claude Code adds extra text to every message a user sends: its own reminders,
and — if a memory plugin such as Hindsight is installed — recalled notes plus
snippets of earlier conversations. Switchyard merges all of that with the real
question and hands the classifier one undivided block. The recalled snippets
quote past questions word for word, so they look exactly like the current one.
Why it matters
In one routing session the classifier averaged 41,225 tokens per call, peaking
at 51,636. Most of that was recalled memory rather than conversation. The real
question is a line or two buried in the middle, and nothing marks it as the one
that matters.
Proposed behavior
Give the injected text its own section, and leave the user's question by itself
at the end:
Route the request in <query>, the last section below. <memory> and <history>
are background: never route on them alone.
<memory>…recalled notes and reminders…</memory>
<history>…recent turns…</history>
<query>…what the user actually typed…</query>
If it all has to be trimmed to fit, cut the memory first and never the question.
Notes
Working locally by splitting on the tags the client already emits
(<system-reminder>, <hindsight_*>) inside
switchyard/lib/processors/llm_classifier/request_processor.py. Happy to open a
PR. Related: #246.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in switchyard/lib/processors/llm_classifier/request_processor.py and inspect how the client’s and <hindsight_*> tags are currently split or passed to the classifier; review related issue #246 for context. Done means injected memory and history are separated from the final user query, the query remains last, and trimming removes memory before the question.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100