NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

[feature] Surface lossy translation diagnostics at runtime

Open
#607 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Problem

Switchyard's translation engine emits structured diagnostics when a provider conversion drops or weakens request data, but the native server path discards those diagnostics. Operators therefore cannot tell when a successful HTTP request was served with a weaker contract than the caller requested.

This is observable with structured output. An OpenAI Chat request containing minLength in its JSON Schema can be routed to an Anthropic Messages target. The Anthropic codec deliberately removes constraints outside its supported subset and emits a lossy_conversion diagnostic. The upstream receives the reduced schema, while the client receives HTTP 200 and neither logs nor /metrics expose that the conversion was lossy.

The repository's translation test already verifies both the schema reduction and the diagnostic. The runtime convenience helpers return only .request, .response, or .body, so the diagnostics do not reach the server or upstream-client observability layers.

Reproduction

Tested on main at 7a72c0667774244d66a8b631e375c9d6e393bf57 using the native server and a request-capture loopback upstream:

  1. Configure an OpenAI Chat ingress route backed by an Anthropic Messages client.

  2. Send a non-streaming request with this response schema:

    {
      "type": "object",
      "properties": {
        "answer": {"type": "string", "minLength": 5}
      },
      "required": ["answer"]
    }
    
  3. Capture the Anthropic request and inspect the client response and /metrics.

Observed:

  • HTTP response status: 200
  • minLength in the captured upstream request: absent
  • response headers naming a translation diagnostic: none
  • metric containing translation: none

The schema reduction is expected provider adaptation. Its invisibility at runtime is the gap.

Proposed solution

Surface existing translation diagnostics at the runtime boundary without changing translation policy:

  • emit one structured warning for each diagnostic, recording only bounded metadata such as severity, diagnostic code, source format, target format, and request/response direction;
  • increment a Prometheus counter for each diagnostic using the same bounded labels;
  • do not include request contents, response contents, JSON paths, model IDs, URLs, or diagnostic message text in metric labels;
  • keep the HTTP response and translated payload unchanged.

A possible metric is:

switchyard_translation_diagnostics_total{severity,code,source,target,direction}

The concrete name and whether severity is useful can follow the project's existing metrics conventions. Source and target should use the three supported wire-format identifiers plus a fixed unknown value when one side is not available.

Alternatives considered

  • Return warnings in response headers. This makes diagnostics request-local, but adds a new client-visible HTTP contract and has awkward size/multiplicity semantics. Logs and metrics are a smaller initial surface.
  • Reject every lossy conversion. LossyConversionPolicy::Reject exists for explicit strict callers, but changing the server default would break currently successful cross-provider requests.
  • Log only. Logs help individual-request debugging, but a counter is needed to detect rates and regressions in production.
  • Metric only. A counter shows frequency but does not give operators a structured event to correlate with a request trace.

Scope notes

  • switchyard-translation continues to own detection and remains transport-independent.
  • Runtime consumers propagate existing diagnostic outputs instead of recomputing lossiness.
  • libsy-llm-client owns outbound request and upstream-response translation telemetry.
  • switchyard-server owns ingress and client-response translation telemetry plus Prometheus initialization/documentation.
  • No translation behavior, public wire response, routing decision, or deployment TOML changes.
  • Tests should cover a lossless request and the minLength reduction through the native HTTP server.

Additional context

Related code and tests:

  • crates/switchyard-translation/src/helpers.rs
  • crates/switchyard-translation/src/engine.rs
  • crates/switchyard-translation/tests/request_translation.rs::openai_schema_constraints_are_removed_from_anthropic_output_format

This also provides an observability foundation for provider-contract questions such as structured-output enforcement, without deciding those translation semantics in this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/switchyard-translation/src/helpers.rs and engine.rs, then trace how existing diagnostic outputs are consumed by libsy-llm-client and switchyard-server. Use the named request translation test as a baseline and add native HTTP server coverage for lossless and lossy requests. Done means bounded structured warnings and Prometheus diagnostics counters are exposed without changing responses or translated payloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, rust
Domain
backend-api-design, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.