Asynchronous or in-loop tool-calling
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Is your feature request related to a problem? Please describe.**
Add native support in NeMo RL for asynchronous or in-loop tool-calling, where the inference engine can autonomously invoke external tools during generation and seamlessly resume once tool responses arrive.
This enables more efficient training with tool-augmented models (e.g., using search, code execution, or retrieval tools) by allowing:
1. Non-blocking generation — other generations can continue while a tool response is pending.
2. Immediate resume — generation automatically continues once a tool result is received.
3. Improved throughput — critical for training with slower tools (responses may take 1–2 minutes).
**Describe the solution you'd like**
Integrate or expose an async tool-calling interface in NeMo RL’s inference/training stack — allowing tool calls to suspend, await responses, and resume generation automatically without blocking other rollouts or samples.
**Describe alternatives you've considered**
vLLM has function calling / tool calling support. You can register functions (tools) with the model and it can generate a function call during inference. vLLM itself does not automatically suspend generation and resume when a tool result is pending. However, its callback-based streaming API allows you to implement this externally.
**Additional context**
This request originated from our engagement with Allen Institute for AI or Ai2's post-training team as part of the Nvidia's partnership with Ai2 ([partnership announcement](https://blogs.nvidia.com/blog/national-science-foundation-ai2-open-ai-models/)).
Contributor guide
Assessment
This issue has not been assessed yet.