NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

feature: Native support for /v1/messages endpoints

Open
#2,226 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement status: needs triage
Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check the docs?
  • I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.

While there is a langchain integration with /v1/messages, there is no external endpoint that accepts /v1/messages payloads. This means it's impossible to directly proxy a /v1/messages endpoint - you'd have to translate your request into /chat/completions first, then translate the response.

I'd like to be able to natively use /v1/messages throughout the library, namely as models specified in the model config (for self-check, etc), or to use the API as my main interface for interacting with NeMo Guardrails.

Describe the solution you'd like
  • A native /v1/messages client, mirroring the existing /v1/chat/completions client, that allows for native, non-langchain communication with /v1/messages models
  • A /v1/messages endpoint in the NeMo Guardrails server, for guardrailed inference proxying
  • A /v1/messages/checks endpoint, for standalone guardrailing
Describe alternatives you've considered

External API translation by some other middleware component. This works, but adds setup complexity and can present issues with streaming.

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing /v1/chat/completions client and NeMo Guardrails server endpoint, then trace how model configuration and standalone checks use them. Define the matching behavior for /v1/messages, /v1/messages/checks, and streaming before implementing the client and proxy paths; done means native payloads work without translation and the existing checks remain available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.