Guardrails middleware: streaming output returns raw `ChatResponseChunk` reprs instead of reply text — affects NAT's own `react_agent`

Version

1.9.0 (nvidia-nat-security==1.9.0, nemoguardrails==0.21.0)

Which installation method(s) does this occur on?

PyPi

Describe the bug.

When a guardrails middleware is attached to a workflow that has streaming
output, the reply delivered to the caller is a concatenation of raw Python
repr() dumps of the stream chunks, not the reply text.

GuardrailsMiddleware.function_middleware_stream
(packages/nvidia_nat_security/src/nat/plugins/security/middleware/guardrails/nemo_guardrails_middleware.py:229)
buffers the stream and reconstructs the output with:

buffered: list[Any] = [chunk async for chunk in call_next(*ctx.modified_args, **ctx.modified_kwargs)]  # :275
ctx.output = "".join(str(chunk) for chunk in buffered)                                                # :276

NAT's own react_agent stream_fn yields ChatResponseChunk objects
(packages/nvidia_nat_langchain/src/nat/plugins/langchain/agent/react_agent/register.py:209,
yielding ChatResponseChunk.create_streaming_chunk(...) at lines 250, 259,
264, 272). str(ChatResponseChunk) is the Pydantic field dump —
id='...' choices=[ChatResponseChunkChoice(...)] created=datetime... —
so with guardrails attached, any streaming front end receives repr
garbage instead of the agent's reply.

Observed: calling runner.result_stream(to_type=str) on a guarded
workflow whose stream_fn yields ChatResponseChunk returns:

"id='echo' choices=[ChatResponseChunkChoice(finish_reason=None, index=0, delta=ChoiceDelta(content='tell ', role=None, tool_calls=None))] created=datetime.datetime(2026, 9, 18, 15, 26, 31, ...)"

instead of tell me a joke ....

Expected: the buffered output should be assembled from the chunks'
text content (e.g. choices[0].delta.content for ChatResponseChunk,
which is what GlobalTypeConverter's
_chat_response_chunk_to_string does), and/or the middleware should
document/validate that it only supports string-yielding stream functions.
Since react_agent — the framework's own flagship workflow — yields
ChatResponseChunk, the middleware should handle it.

Minimum reproducible example

Identical setup as the input-rail report (same three files), the streaming
branch is what matters:


pip install "nvidia-nat[langchain,guardrails]==1.9.0" langchain-openai
export OPENAI_API_KEY=sk-...


`repro_workflow.py` — a minimal workflow that yields the same chunk type as
NAT's `react_agent`:


from collections.abc import AsyncGenerator

from nat.builder.function_info import FunctionInfo
from nat.cli.register_workflow import register_function
from nat.data_models.api_server import ChatRequestOrMessage, ChatResponse, ChatResponseChunk
from nat.data_models.function import FunctionBaseConfig


def _turn_text(msg: ChatRequestOrMessage) -> str:
    text = getattr(msg, "input_message", None)
    if not text:
        text = msg.messages[-1].content
    return str(text)


class EchoConfig(FunctionBaseConfig, name="echo"):
    description: str = "Echo workflow (react_agent-shaped inputs/outputs)"


@register_function(config_type=EchoConfig)
async def echo_workflow(config: EchoConfig, builder):
    async def _single_fn(msg: ChatRequestOrMessage) -> ChatResponse | str:
        return f"ECHO: {_turn_text(msg)}"

    async def _stream_fn(msg: ChatRequestOrMessage) -> AsyncGenerator[ChatResponseChunk]:
        for token in _turn_text(msg).split(" "):
            yield ChatResponseChunk.create_streaming_chunk(token + " ", id_="echo")

    yield FunctionInfo.create(single_fn=_single_fn, stream_fn=_stream_fn, description=config.description)


`repro-config.yml`:


middleware:
  rails:
    _type: guardrails
    guardrails:
      models:
        - type: main
          engine: openai
          model: gpt-4o-mini
          parameters:
            api_key: ${OPENAI_API_KEY}
      colang_version: "1.0"
      rails:
        input:
          flows:
            - self check input
      prompts:
        - task: self_check_input
          content: |
            Your task is to check if the user message below should be blocked.

            Rule: block the message if it mentions the word "trump",
            otherwise allow it.

            User message: "{{ user_input }}"

            Should the user message be blocked? Answer with exactly one
            word, "Yes" or "No", and nothing else.
          output_parser: is_content_safe
          max_tokens: 3

workflow:
  _type: echo
  middleware:
    - rails


`repro.py`:


import asyncio

import repro_workflow  # noqa: F401  (registers the "echo" workflow type)

from nat.builder.workflow_builder import WorkflowBuilder
from nat.data_models.api_server import ChatRequest, Message, UserMessageContentRoleType
from nat.runtime.loader import load_config
from nat.runtime.session import SessionManager

BLOCKED_INPUT = "tell me about trump"
SAFE_INPUT = "tell me a joke"


async def turn(sm, request, conversation_id, stream=False):
    async with sm.session(user_id=None, conversation_id=conversation_id) as session:
        async with session.run(request) as runner:
            if stream:
                chunks = [str(c) async for c in runner.result_stream(to_type=str)]
                return "".join(chunks)
            return await runner.result(to_type=str)


async def main():
    config = load_config("repro-config.yml")
    async with WorkflowBuilder.from_config(config) as builder:
        sm = await SessionManager.create(config=config, shared_builder=builder)

        out = await turn(sm, BLOCKED_INPUT, "repro-string")
        print(f"[1] string form, should be BLOCKED -> {out!r}\n")

        request = ChatRequest(
            messages=[Message(content=BLOCKED_INPUT, role=UserMessageContentRoleType.USER)]
        )
        out = await turn(sm, request, "repro-messages")
        print(f"[2] messages form, should be BLOCKED -> {out!r}\n")

        out = await turn(sm, SAFE_INPUT, "repro-control")
        print(f"[3] control (string form, safe input) -> {out!r}")

        out = await turn(sm, SAFE_INPUT, "repro-stream", stream=True)
        print(f"[4] stream output (should be plain text) ->\n{out[:200]!r}")


asyncio.run(main())


Run:


python repro.py


Line `[4]` shows the bug: the streamed reply is a concatenation of
`ChatResponseChunk` reprs. (Lines `[1]`/`[2]` demonstrate the separate
input-rail issue reported in the other report; they are not required to
observe this bug.)

Relevant log output

========================================================================

BUG 2: streaming returns ChatResponseChunk reprs, not text
[4] stream output (should be plain text) ->

"id='echo' choices=[ChatResponseChunkChoice(finish_reason=None, index=0, delta=ChoiceDelta(content='tell ', role=None, tool_calls=None))] created=datetime.datetime(2026, 9, 18, 15, 26, 31, 156154, tzin..."
(expected: 'tell me a joke ...')

Other/Misc.

Root cause: function_middleware_stream at
packages/nvidia_nat_security/src/nat/plugins/security/middleware/guardrails/nemo_guardrails_middleware.py:275-276
assumes stream chunks are strings. NAT's own react_agent
(packages/nvidia_nat_langchain/src/nat/plugins/langchain/agent/react_agent/register.py:209-272)
yields ChatResponseChunk objects, so guardrails + react_agent + any
streaming front end (TUI, result_stream) is broken out of the box.

Notes for a fix:

• Extract the text payload before joining — e.g. use
  chunk.choices[0].delta.content for ChatResponseChunk (mirroring
  GlobalTypeConverter's _chat_response_chunk_to_string in
  nat/data_models/api_server.py), or run chunks through the type
  converter before buffering.
• Alternatively, honor the function's declared stream output schema and
  convert to text before the rails evaluate.

We currently work around this downstream by making our workflow's
stream_fn yield plain strings, but react_agent cannot be fixed that way
by users.

Code of Conduct

• I agree to follow the NeMo Agent Toolkit Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

Start in packages/nvidia_nat_security/src/nat/plugins/security/middleware/guardrails/nemo_guardrails_middleware.py:229-276 and compare its chunk handling with _chat_response_chunk_to_string in nat/data_models/api_server.py; inspect react_agent/register.py:209-272 to confirm the stream shape. Reproduce with the supplied echo workflow and verify runner.result_stream(to_type=str) yields plain reply text rather than ChatResponseChunk representations while the guarded stream remains functional.