NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
Neutral IR does not record structured-output enforcement
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Summary
The neutral request IR records what schema a caller wants but not whether the provider must enforce it. Enforcement is a first-class part of the contract on both providers we translate between, and it is expressed differently on each side, so it is lost or invented at the boundary.
Split out of #462 at a maintainer's request to keep that PR in limited scope.
The gap
OutputParams::response_format holds an OpenAI-shaped response_format value. Enforcement lives inside that value as json_schema.strict, which makes it a property of one provider's wire encoding rather than something the IR states.
The two providers do not agree on how enforcement is expressed:
- Anthropic —
output_config.formatwithtype: json_schemais enforced. There is no flag; supplying a schema is the request for enforcement. - OpenAI Chat — a schema is advisory unless
strict: trueis set. Their guide is explicit that "only Structured Outputs ensure schema adherence"; without the flag you get valid JSON with no guarantee it matches.
So the two directions fail differently:
- Anthropic → OpenAI Chat loses enforcement. The schema is forwarded, the guarantee is not, and the caller can receive prose where they asked for JSON. This is the case reported in #452.
- OpenAI Chat → Anthropic gains it.
encode_anthropic_output_formatkeeps only the schema, so a request that arrived withstrictabsent is translated into an Anthropic request that is enforced anyway.
Neither is visible to the caller, and neither produces a diagnostic today.
Why it is not a one-line fix
Setting strict: true whenever a schema is present would trade a quiet degradation for a loud regression, because the accepted schema subsets differ:
Anthropic output_config.format |
OpenAI strict: true |
|
|---|---|---|
every property listed in required |
not required — optional properties are allowed | required |
additionalProperties: false |
required | required |
allOf |
supported, with limits | not allowed |
string format, minItems |
supported | restricted for fine-tuned models |
The first row is the one that bites. An Anthropic schema with any optional property is legal today, and OpenAI rejects an out-of-subset schema at request time when strict is on — so a request that currently returns unenforced output would start returning 400 instead.
This codec already handles the mirror-image problem rather than passing schemas through and hoping: strip_anthropic_unsupported_constraints narrows an OpenAI schema to what Anthropic accepts when translating the other way.
Possible directions
Listed for discussion, not as a proposal:
- Record enforcement in the IR — an explicit field on
OutputParamsso each codec decides how to express it, rather than encoding one provider's spelling and reinterpreting it elsewhere. Largest change, and the one that actually removes the ambiguity. - Derive
strictfrom the schema — emitstrict: trueonly when the schema provably satisfies the strict subset, and omit it otherwise. No IR change, enforcement where it is safe, and never turns a working request into a 400. Does not help the schemas that fall outside the subset. - Diagnose the downgrade — leave the behavior alone and emit a lossiness diagnostic when enforcement is dropped. Note that under
LossyConversionPolicy::Rejectthis would fail every affected request, so it likely needs its own severity rather than reusingpush_lossy.
(1) and (2) are not exclusive; (2) is a reasonable step even if (1) lands later.
Related
- #452 — the reported symptom, an Anthropic-ingress schema reaching the upstream unenforced.
- #462 — restores the schema itself; deliberately does not touch enforcement.
Environment
main at f9b2df61.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing OutputParams::response_format through the Anthropic and OpenAI Chat translation paths, including encode_anthropic_output_format and strip_anthropic_unsupported_constraints. Read related issues #452 and #462 to understand the existing schema behavior and scope. Done requires an agreed enforcement representation or handling strategy, with both translation directions and lossiness behavior covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100