feature: Support tools param on /v1/chat/completions

Open
#2,352 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ai, api, backend

Research direction

Start at the /v1/chat/completions server entry point and inspect the existing non-streaming tool-calling support added in #1942. Review related work in #2056, #1615, and #2268, then trace request forwarding, response serialization, and streaming behavior. Done means supported tool parameters and tool_calls follow the OpenAI spec with rails applied, or unsupported requests return a clear 422.

Written by the indexing model from the issue text.

Description

enhancement status: needs triage
Did you check the docs?
  • I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.

Requests to /v1/chat/completions containing the standard OpenAI tools parameter
return HTTP 200, but:

  1. tools / tool_choice are silently dropped — no warning, no error, and the model
    responds with plain text as if no tools were provided.
  2. message.tool_calls is never present in the response — tool calls are stripped
    even when the upstream model emits them.

There is no way to tell from the client side that tool calling isn't supported —
requests just quietly degrade to text-only responses. This makes the endpoint
unusable as a drop-in guardrails proxy for tool-calling agents, which is one of the
main advertised use cases (agentic security, tool input/output rails).

Tool calling params were added to the server in 0.23.0 (#1942), but limited to
`non-streaming. Related open work: #1615, #2056, #2268.

Describe the solution you'd like
  1. /v1/chat/completions accepts tools, tool_choice, and parallel_tool_calls,
    forwards them to the model, and returns message.tool_calls with
    finish_reason: "tool_calls" per the OpenAI spec — with tool input/output rails
    applied.
  2. Streaming support (per #2056), since most agent frameworks and clients stream by
    default.
  3. Where unsupported, reject requests containing tools with a 422 and a clear error
    message instead of silently dropping them.
Describe alternatives you've considered
  • Calling the LLM directly and using the checks endpoint for input/output validation
  • Disabling streaming in our client
Additional context

No response

Dominant language
Python
Stars
7.2k
Forks
843
Avg merge
3d 1h
Merged PRs (30d)
25

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-NeMo/Guardrails

All issues in NVIDIA-NeMo/Guardrails

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.