NVIDIA-NeMo / NVIDIA-NeMo/Guardrails

bug: nemoguardrails server api streaming is not working

Open
#893 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
7.2k
Forks
842
Avg merge
3d 1h
Merged PRs (30d)
25

Description

Did you check docs and existing issues?
  • I have read all the NeMo-Guardrails docs
  • I have updated the package to the latest version before submitting this issue
  • (optional) I have used the develop branch
  • I have searched the existing issues of NeMo-Guardrails
Python version (python --version)

3.11

Operating system/version

15

NeMo-Guardrails version (if you must use a specific version and not the latest

0.11.0

Describe the bug

I started a server with the following command, but streaming mode is not working

python -m nemoguardrails server --config=./config --default-config-id=config

config.yml

models:
  - type: main
    engine: openai
    model: gpt-4o-mini
Steps To Reproduce

my test code

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()

client = OpenAI(
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Hello"},
    ],
    stream=True,
    extra_body={
        "config_id": "config"
    }
)

for chunk in response:
    print(chunk.choices[0].delta.content)
Expected Behavior

print stream chunks

Actual Behavior

print nothing

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running python -m nemoguardrails server --config=./config --default-config-id=config with the provided config and OpenAI client example. Trace the server's streaming response path and compare it with the non-streaming behavior. Done means the client receives and prints response chunks; the issue does not name a source file or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.