NVIDIA-NeMo / NVIDIA-NeMo/RL

Asynchronous or in-loop tool-calling

Open
#1,387 6 comments 0 reactions 1 assignee Claimed by @parthchadha View on GitHub
enhancement example x-a21
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Is your feature request related to a problem? Please describe.**
Add native support in NeMo RL for asynchronous or in-loop tool-calling, where the inference engine can autonomously invoke external tools during generation and seamlessly resume once tool responses arrive.

This enables more efficient training with tool-augmented models (e.g., using search, code execution, or retrieval tools) by allowing:
1. Non-blocking generation — other generations can continue while a tool response is pending.
2. Immediate resume — generation automatically continues once a tool result is received.
3. Improved throughput — critical for training with slower tools (responses may take 1–2 minutes).

**Describe the solution you'd like**
Integrate or expose an async tool-calling interface in NeMo RL’s inference/training stack — allowing tool calls to suspend, await responses, and resume generation automatically without blocking other rollouts or samples.

**Describe alternatives you've considered**
vLLM has function calling / tool calling support. You can register functions (tools) with the model and it can generate a function call during inference. vLLM itself does not automatically suspend generation and resume when a tool result is pending. However, its callback-based streaming API allows you to implement this externally.

**Additional context**
This request originated from our engagement with Allen Institute for AI or Ai2's post-training team as part of the Nvidia's partnership with Ai2 ([partnership announcement](https://blogs.nvidia.com/blog/national-science-foundation-ai2-open-ai-models/)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.