NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

epic: close OpenAI API conformance gaps in Guardrails server

Open
#2,249 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Goal

The NeMo Guardrails server's OpenAI-compatible API should match the
CreateChatCompletionRequest / CreateChatCompletionResponse / ListModels
contracts closely enough that standard OpenAI SDK clients work without
surprises. This is an important step for this server to become a transparent inference proxu

Why

The conformance CI (api-conformance-impl.yml) reports around 100 gaps between
app.openapi() and OpenAI v2.3.0. Some are harmless schema metadata diffs, but
others cause real problems like missing request properties, route metadata differences etc

Scope

Potential sub-issues might include:

  1. Missing request properties (~23 gaps)
  2. Request schema typing — nullable, defaults, anyOf/oneOf (~24 gaps)
  3. Response usage required fields (~7 gaps)
  4. Response types, enums, nullables, streaming media type (~24 gaps)
  5. Route metadata — operation IDs, tags (4 gaps)
  6. GET /models response schema (~9 gaps)
  7. Conformance script noise reduction (~10 gaps)

Local reproduction (requires oasdiff):

# Install oasdiff e.g.
brew install oasdiff

# Run the conformance check (fetches the OpenAI spec and diffs via oasdiff)
uv run --locked python scripts/openai_coverage.py --fetch --fastapi

Completion criteria

  • scripts/openai_coverage.py --fastapi reports 0 conformance gaps (excluding
    the intentional-extensions allowlist).
  • Intentional Guardrails extensions (guardrails request/response properties,
    422 status) are reported separately, not as gaps.
  • Unimplemented endpoints (Stored Completions, single-model CRUD) are excluded
    from the gap count.
  • All existing tests pass; new tests cover the added request properties and
    response schema changes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/openai_coverage.py and run uv run --locked python scripts/openai_coverage.py --fetch --fastapi to review the reported gaps against the OpenAI v2.3.0 contracts. Work through the listed request, response, route, and model-schema differences, then verify that the command reports no unallowlisted gaps and that all existing and new tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, openapi, python
Domain
api, backend, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.