NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[bug]: --dry-run accepts configured HTTP headers that cannot be sent
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Symptom
switchyard-server --dry-run accepts configured HTTP headers that cannot be used to build an upstream request. The server also starts normally, but every request routed through the affected client returns HTTP 502 before reaching the upstream.
Reproduction
No provider credentials or external network access are required. Start a loopback HTTP server on port 19450, then save this configuration as routes.toml:
schema_version = 1
[llm_clients.upstream]
format = "openai_chat"
base_url = "http://127.0.0.1:19450/v1"
extra_headers = { "bad header" = "value" }
max_retries = 0
[targets.upstream]
id = "upstream/model"
llm_client = "upstream"
[routes.default]
id = "switchyard/default"
type = "passthrough"
target = "upstream"
Run the deployment check:
cargo run --locked -p switchyard-server -- --config routes.toml --dry-run
It exits with status 0 and prints:
server OK: switchyard/default
Start the same configuration:
cargo run --locked -p switchyard-server -- \
--config routes.toml --host 127.0.0.1 --port 19451
Send a request:
curl -i http://127.0.0.1:19451/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"switchyard/default","messages":[{"role":"user","content":"hello"}]}'
The response is:
HTTP/1.1 502 Bad Gateway
{"error":{"message":"failed to build upstream request: builder error","type":"upstream_error","code":"upstream_configuration_error"}}
The loopback upstream receives no request. A valid control using
extra_headers = { "x-audit-header" = "valid-control" } reaches the same upstream and returns HTTP 200.
The same startup/runtime mismatch occurs when api_key_env resolves to a value containing an HTTP control character, such as a newline: --dry-run exits successfully, but the routed request returns the same HTTP 502 without reaching the upstream.
Expected vs. actual
- Expected: Client construction and
--dry-runreject header names, configured header values, and generated authentication header values that the HTTP client cannot send. The error should identify the affected client and field without exposing secrets. - Actual: Validation checks conflicts with reserved headers, but header construction errors are deferred until the first routed request.
Environment
- Switchyard commit SHA:
4eae4bf1e464bd778f1e68c4d5824bc195c3e692 - Python version: N/A for the server path; Python standard library used only for the loopback capture server
- Rust version:
rustc 1.96.1 (31fca3adb 2026-06-26) - OS / arch: macOS 26.5.2 / arm64
- Install path: source build with
cargo build -p switchyard-server - Inbound format: OpenAI Chat Completions
- Backend: offline loopback OpenAI-compatible server
Additional context
The behavior was reproduced with the compiled server as a standalone process and real TCP requests. Invalid header-name and invalid API-key cases failed before the loopback server recorded a request; the valid-header control recorded the rewritten model and configured header.
Related work does not cover this validation gap:
- #398 added early validation for malformed upstream URLs after the same dry-run/runtime mismatch.
- #360 rejects reserved authentication and protocol names in
extra_headers, but does not validate whether names and values are constructible.
A startup check should use the same header conversion semantics as request construction and remain entirely offline.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at switchyard-server’s client-construction path and compare --dry-run validation with the request header-construction path, including extra_headers and api_key_env. Done means invalid names and values fail offline with client and field context without exposing secrets, while valid headers and routed requests continue to work; add or run regression coverage for both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100