NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
feat(libsy-llm-client): report rejected extra_body keys as a configuration error
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Problem
Per-target extra_body keys are merged into the outbound request body in
crates/libsy-llm-client/src/client.rs at a point that sits outside the retry loop, so the merge
happens once, before any attempt.
When an upstream rejects the request because of one of those keys — a 400/422 naming an unknown or
unexpected parameter — every retry re-sends the same rejected key. The retry cannot succeed, and
there is no way for a caller to say "drop this knob and try again".
The case we hit in practice: reasoning-control knobs (chat_template_kwargs, enable_thinking) that
some OpenAI-compatible servers accept and others reject outright. A call carrying the knob fails
permanently against a rejecting server, even though the identical call without it would succeed.
Proposal
Move the extra_body merge inside the retry loop and allow named keys to be dropped after a
parameter rejection. Roughly 20 lines.
Deliberately narrow:
- Only a 400/422 whose body names the parameter, or an unknown/extra-parameter condition,
qualifies. A 429 or 5xx must surface as-is — silently stripping a knob and re-firing on a
rate limit would double the load and mask the real failure. - Retry exactly once without the knob. No escalation ladder, no second strip.
Scope
crates/libsy-llm-client/ only. Benefits every route that configures extra_body, which is why it
seems worth doing as its own change rather than inside a feature PR.
Notes
Happy to open the PR. Wanted to check the direction first, since it moves an existing statement
across the retry boundary rather than adding alongside it — if you'd prefer a different shape (e.g.
an explicit per-target "droppable keys" list instead of inferring from the reject), that's easy to
adjust.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in crates/libsy-llm-client/src/client.rs, locate the per-target extra_body merge and the retry loop, then trace how upstream 400/422, 429, and 5xx responses are classified. Done means a named-parameter rejection retries exactly once without that key, while rate-limit and server errors are surfaced unchanged; keep the change within crates/libsy-llm-client/.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100