NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[bug] Keep upstream error content out of logs and traces
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Symptom
Upstream errors can contain provider response text. Providers sometimes quote request content in validation errors, so formatting the full error into a log or trace can expose prompts, source code, or other request data.
The handling is currently inconsistent. #611 removes this content from the inner libsy.client_call, libsy.llm_call, and libsy.run records, but the server request log, Advisor fail-open audit, and SSE error path can still record the raw message.
Expected vs. actual
- Expected: Keep the complete typed error available for retries, fallback, and the response returned to the caller. Logs and traces should contain only safe structured fields such as the error kind, status, target, and phase.
- Actual: Several paths use the error's
Displayoutput, which can include the upstream body or another free-form message. Other paths remove all error detail, including useful operational information.
Scope notes
Please settle the common logging behavior and apply it consistently to:
libsy.client_call,libsy.llm_call, andlibsy.run- terminal
switchyard_server::requestevents - Advisor fail-open warnings and audit events
- server SSE errors
- runner and Relay failure reporting
The original error should remain unchanged in flight. Avoid adding another public formatting API unless an external integration needs it. Add regression tests at the final log and trace sinks, not only the inner client span.
#536 and #537 added the typed RouteErrorSummary used by embedded runner integrations. This work should align with that shape where it fits rather than introduce a second string-based contract.
Additional context
- #610 exposed the original upstream-body case during review.
- #611 is a partial implementation and reproduction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the logging and tracing sinks for libsy.client_call, libsy.llm_call, libsy.run, switchyard_server::request, Advisor fail-open events, server SSE errors, and runner and Relay failure reporting. Read #611 and the typed RouteErrorSummary from #536 and #537 first. Done means the original typed error remains available while every listed sink records only safe structured fields, with regression tests at the final sinks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, observability, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100