NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

[bug] Keep upstream error content out of logs and traces

Open
#615 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Symptom

Upstream errors can contain provider response text. Providers sometimes quote request content in validation errors, so formatting the full error into a log or trace can expose prompts, source code, or other request data.

The handling is currently inconsistent. #611 removes this content from the inner libsy.client_call, libsy.llm_call, and libsy.run records, but the server request log, Advisor fail-open audit, and SSE error path can still record the raw message.

Expected vs. actual

  • Expected: Keep the complete typed error available for retries, fallback, and the response returned to the caller. Logs and traces should contain only safe structured fields such as the error kind, status, target, and phase.
  • Actual: Several paths use the error's Display output, which can include the upstream body or another free-form message. Other paths remove all error detail, including useful operational information.

Scope notes

Please settle the common logging behavior and apply it consistently to:

  • libsy.client_call, libsy.llm_call, and libsy.run
  • terminal switchyard_server::request events
  • Advisor fail-open warnings and audit events
  • server SSE errors
  • runner and Relay failure reporting

The original error should remain unchanged in flight. Avoid adding another public formatting API unless an external integration needs it. Add regression tests at the final log and trace sinks, not only the inner client span.

#536 and #537 added the typed RouteErrorSummary used by embedded runner integrations. This work should align with that shape where it fits rather than introduce a second string-based contract.

Additional context

  • #610 exposed the original upstream-body case during review.
  • #611 is a partial implementation and reproduction.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the logging and tracing sinks for libsy.client_call, libsy.llm_call, libsy.run, switchyard_server::request, Advisor fail-open events, server SSE errors, and runner and Relay failure reporting. Read #611 and the typed RouteErrorSummary from #536 and #537 first. Done means the original typed error remains available while every listed sink records only safe structured fields, with regression tests at the final sinks.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.