NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

feature: multi-message evaluation support in /v1/checks

Open
#2,277 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement status: needs triage
Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check the docs?
  • I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.

The /v1/checks endpoint (check_async) only evaluates the last user message in the messages array. When a client sends a chat completion request with conversation history containing multiple user messages, only the final one is checked against the configured rails.

This is a problem for gateway/proxy integrations (such as ai-gateway-payload-processing and Praxis) that need to validate all user messages in a request before forwarding it to the model.

The single-message limitation forces a workaround: splitting the request into N individual HTTP calls, one per user message.

I benchmarked this workaround against TrustyAI NeMo-Guardrails (develop HEAD, July 16) with PII detection rails and 1ms simulated pod-to-pod RTT:

User messages | Single call | N sequential calls | Overhead
        1     |       39ms  |              26ms  |     ~0%
        5     |      142ms  |             133ms  |     ~0%
       10     |      223ms  |             309ms  |    +39%
       20     |      437ms  |             597ms  |    +37% (+160ms)

At 10+ messages (typical for conversation history), the workaround adds ~37% latency purely from HTTP round-trip overhead.

Describe the solution you'd like

Add multi-message evaluation support to /v1/checks so that all user messages in the array are individually checked against the configured rails, not just the last one. The response should reflect the aggregate result:

  • passed if all messages passed
  • blocked if any message was blocked (short-circuit)
  • modified if any message was modified (with the transformed content)

This would let integrators use a single call to get both full message coverage and redaction support.

Describe alternatives you've considered

Split into N sequential HTTP calls - works but adds ~37% latency at 10+ messages and increases code complexity.

Additional context

Benchmark script

nemo-latency.zip

cc @m-misiura @christinaexyou @RobGeada @tgasser-nv @Pouyanpi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Python implementation of the /v1/checks endpoint and its check_async entry point. Trace how the messages array is reduced to the last user message and inspect the existing response handling. Done means all user messages are evaluated, blocked results short-circuit, modified content is returned, and passed is reported only when all messages pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.