NVIDIA-NeMo / NVIDIA-NeMo/Switchyard

[feature] Define JSON Object contracts for custom classifiers

Open
#429 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
291
Avg merge
1d 8h
Merged PRs (30d)
182

Description

Problem

#411 adds JSON Object mode to the packaged capability and escalation classifiers. Those classifiers always have a packaged verdict schema, so Switchyard can append it to the prompt and validate the returned object locally.

Custom classifiers differ: the user owns both the prompt and the inner response_schema. Exposing response_format_type = "json_object" for custom mode requires a clear contract for these cases:

  • response_schema is omitted.
  • The prompt already contains a hard-coded copy of the schema or other JSON output instructions.
  • response_schema contains a provider response-format wrapper such as {"type": "json_object"} instead of an inner JSON Schema.
  • The prompt and configured schema disagree.

Without an explicit policy, Switchyard could duplicate or contradict prompt instructions, or accept verdicts without the local validation used by the packaged classifiers.

Proposed solution

Define the custom-classifier contract before exposing JSON Object mode. A conservative design would:

  • continue to require an inner response_schema in both response modes;
  • append that schema to the prompt and validate verdicts locally in json_object mode;
  • reject provider response-format wrappers where an inner JSON Schema is expected; and
  • document that the configured schema, rather than a hard-coded prompt copy, is the source of truth.

The design should explicitly decide whether schema-free JSON Object output is unsupported or a separate, unvalidated mode.

Alternatives considered

  • Request JSON Object output without a schema. This preserves JSON syntax but loses the contract needed for local validation and deterministic fallback.
  • Treat a schema hard-coded in the prompt as authoritative. Switchyard cannot reliably extract or validate arbitrary prompt text, and it can drift from response_schema.

Scope notes

  • ClassifierContract::from_inner_schema and CustomClassifierConfig in libsy.
  • The server llm_classifier custom-mode TOML surface.
  • Python bindings if the response-format option is exposed there.
  • Keep existing custom classifiers on JSON Schema by default.

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ClassifierContract::from_inner_schema and CustomClassifierConfig in libsy, then inspect the server llm_classifier custom-mode TOML surface and any Python binding for response-format exposure. Done means the custom-classifier JSON Object contract explicitly covers missing schemas, prompt/schema disagreement, provider wrappers, local validation, and the default JSON Schema behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
json, python, rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.