NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

feat(libsy-llm-client): report rejected extra_body keys as a configuration error

Open
#260 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Problem

Per-target extra_body keys are merged into the outbound request body in
crates/libsy-llm-client/src/client.rs at a point that sits outside the retry loop, so the merge
happens once, before any attempt.

When an upstream rejects the request because of one of those keys — a 400/422 naming an unknown or
unexpected parameter — every retry re-sends the same rejected key. The retry cannot succeed, and
there is no way for a caller to say "drop this knob and try again".

The case we hit in practice: reasoning-control knobs (chat_template_kwargs, enable_thinking) that
some OpenAI-compatible servers accept and others reject outright. A call carrying the knob fails
permanently against a rejecting server, even though the identical call without it would succeed.

Proposal

Move the extra_body merge inside the retry loop and allow named keys to be dropped after a
parameter rejection. Roughly 20 lines.

Deliberately narrow:

  • Only a 400/422 whose body names the parameter, or an unknown/extra-parameter condition,
    qualifies. A 429 or 5xx must surface as-is — silently stripping a knob and re-firing on a
    rate limit would double the load and mask the real failure.
  • Retry exactly once without the knob. No escalation ladder, no second strip.
Scope

crates/libsy-llm-client/ only. Benefits every route that configures extra_body, which is why it
seems worth doing as its own change rather than inside a feature PR.

Notes

Happy to open the PR. Wanted to check the direction first, since it moves an existing statement
across the retry boundary rather than adding alongside it — if you'd prefer a different shape (e.g.
an explicit per-target "droppable keys" list instead of inferring from the reject), that's easy to
adjust.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/libsy-llm-client/src/client.rs, locate the per-target extra_body merge and the retry loop, then trace how upstream 400/422, 429, and 5xx responses are classified. Done means a named-parameter rejection retries exactly once without that key, while rate-limit and server errors are surfaced unchanged; keep the change within crates/libsy-llm-client/.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.