NVIDIA-NeMo / NVIDIA-NeMo/Switchyard
[feature] Define JSON Object contracts for custom classifiers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 291
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 182
Description
Problem
#411 adds JSON Object mode to the packaged capability and escalation classifiers. Those classifiers always have a packaged verdict schema, so Switchyard can append it to the prompt and validate the returned object locally.
Custom classifiers differ: the user owns both the prompt and the inner response_schema. Exposing response_format_type = "json_object" for custom mode requires a clear contract for these cases:
response_schemais omitted.- The prompt already contains a hard-coded copy of the schema or other JSON output instructions.
response_schemacontains a provider response-format wrapper such as{"type": "json_object"}instead of an inner JSON Schema.- The prompt and configured schema disagree.
Without an explicit policy, Switchyard could duplicate or contradict prompt instructions, or accept verdicts without the local validation used by the packaged classifiers.
Proposed solution
Define the custom-classifier contract before exposing JSON Object mode. A conservative design would:
- continue to require an inner
response_schemain both response modes; - append that schema to the prompt and validate verdicts locally in
json_objectmode; - reject provider response-format wrappers where an inner JSON Schema is expected; and
- document that the configured schema, rather than a hard-coded prompt copy, is the source of truth.
The design should explicitly decide whether schema-free JSON Object output is unsupported or a separate, unvalidated mode.
Alternatives considered
- Request JSON Object output without a schema. This preserves JSON syntax but loses the contract needed for local validation and deterministic fallback.
- Treat a schema hard-coded in the prompt as authoritative. Switchyard cannot reliably extract or validate arbitrary prompt text, and it can drift from
response_schema.
Scope notes
ClassifierContract::from_inner_schemaandCustomClassifierConfigin libsy.- The server
llm_classifiercustom-mode TOML surface. - Python bindings if the response-format option is exposed there.
- Keep existing custom classifiers on JSON Schema by default.
Additional context
- Follow-up to #411.
- Raised in the #411 review: https://github.com/NVIDIA-NeMo/Switchyard/pull/411#issuecomment-5296313660
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ClassifierContract::from_inner_schema and CustomClassifierConfig in libsy, then inspect the server llm_classifier custom-mode TOML surface and any Python binding for response-format exposure. Done means the custom-classifier JSON Object contract explicitly covers missing schemas, prompt/schema disagreement, provider wrappers, local validation, and the default JSON Schema behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json, python, rust
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100