bug: jailbreak detection ONNX model classifies benign prompts as jailbreaks

Did you check docs and existing issues?

• I have read all the NeMo-Guardrails docs
• I have updated the package to the latest version before submitting this issue
• (optional) I have used the develop branch
• I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

pyhton 3.12.14

Operating system/version

Ubuntu 26.04 LTS

NeMo-Guardrails version (if you must use a specific version and not the latest

0.24.0

Describe the bug

The jailbreak detection model rail produces incorrect classifications when using the published nvidia/NemoGuard-JailbreakDetect ONNX artifact.

The same 768-dimensional embeddings produce different classifications when passed to the published snowflake.pkl and snowflake.onnx classifier artifacts.

The NeMo SnowflakeEmbed implementation itself does not appear to be the source of the discrepancy: its output was verified to be exactly identical to the embedding produced directly using AutoTokenizer and AutoModel from the same Snowflake/snowflake-arctic-embed-m-long model snapshot.

However, the ONNX classifier returns jailbreak=True for ordinary benign prompts such as Hello, What is Python?, Explain Docker, and Fix Python syntax.

In contrast, the published sklearn PKL classifier classifies the same embeddings as class 0 (non-jailbreak).

This makes the ONNX-based jailbreak detection path unusable as currently observed, since benign prompts are incorrectly detected as jailbreaks.

Steps To Reproduce

1. Install NeMo Guardrails 0.24.0.

2. Configure the jailbreak detection rail to use the model-based jailbreak detector.

3. Use the official Snowflake/snowflake-arctic-embed-m-long embedding model.

4. Load the published nvidia/NemoGuard-JailbreakDetect artifacts.

5. Generate embeddings for ordinary benign prompts such as:

Hello
What is Python?
Explain Docker
Fix Python syntax

6. Run the same embeddings through the published snowflake.pkl classifier using scikit-learn.

Observed PKL results include:

Hello               -> class 0, P(class=1)=0.0085
What is Python?     -> class 0, P(class=1)=0.0171
Write factorial     -> class 0, P(class=1)=0.0166

7. Run the same embeddings through the published snowflake.onnx classifier using the NeMo inference implementation.

Observed ONNX results include:

Hello               -> class 1
What is Python?     -> class 1
Explain Docker      -> class 1
Fix Python syntax   -> class 1

The discrepancy is reproducible using the same embedding vectors.

Expected Behavior

The ONNX classifier should produce classifications consistent with the published snowflake.pkl classifier when both receive the same 768-dimensional embedding.

Ordinary benign prompts should not be classified as jailbreaks.

Actual Behavior

The published snowflake.onnx classifier returns class 1 / jailbreak=True for ordinary benign prompts.

The same embeddings are classified as class 0 by the published snowflake.pkl classifier.

For example:

Input: Hello
PKL:  class 0, P(class=1)=0.0085
ONNX: class 1

Input: What is Python?
PKL:  class 0, P(class=1)=0.0171
ONNX: class 1

Input: Explain Docker
ONNX: class 1

Input: Fix Python syntax
ONNX: class 1

The NeMo embedding implementation was independently compared with direct inference using the same Snowflake model and produced identical 768-dimensional embeddings, so the observed discrepancy appears to occur at the classifier artifact or ONNX inference stage rather than at the embedding stage.

Start by reproducing the discrepancy in the NeMo inference implementation with the published snowflake.pkl and snowflake.onnx artifacts and identical 768-dimensional embeddings. Compare the classifier outputs for the listed benign prompts; done means the ONNX artifact produces classifications consistent with the PKL classifier and does not flag those prompts as jailbreaks.