AsyncGRPO: support async tool calls in AsyncRolloutWorker
- Dominant language
- Python
- Stars
- 19.3k
- Forks
- 3k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 194
Description
### Feature request
## Feature request
GRPOTrainer already supports async tools, but trl.experimental.async_grpo.AsyncGRPOTrainer currently does not.
At the moment, AsyncRolloutWorker rejects coroutine tools during initialization with:
ValueError("Asynchronous tools are not supported in AsyncRolloutWorker yet.")
This creates a feature gap between the mainline and experimental GRPO implementations.
I would like to propose adding async tool support to trl.experimental.async_grpo.async_rollout_worker.AsyncRolloutWorker with a minimal parity patch.
Relevant code:
- Mainline async tool support in GRPOTrainer
- Experimental async trainer in trl.experimental.async_grpo.AsyncGRPOTrainer
Related merged PR:
- PR #4742 added async tool calls to mainline GRPOTrainer
No paper is directly required for this proposal since this is a feature-parity / implementation-gap request rather than a new algorithm proposal.
### Motivation
## Motivation
Many agent tools are naturally I/O-bound, for example:
- external APIs
- retrieval
- browser / sandbox environments
- RPC-backed tools
In these cases, forcing tools to be synchronous serializes a latency-heavy part of the rollout pipeline.
Since GRPOTrainer already supports async tools, it is surprising that AsyncGRPOTrainer still rejects them. This makes it harder to use AsyncGRPOTrainer for realistic
agent training settings, even though it already supports:
- async reward functions
- tool use
- environment_factory
So the motivation is mainly:
- close the feature gap with mainline GRPOTrainer
- make async agent environments usable in AsyncGRPOTrainer
- do so with minimal scope and without changing the public API
### Your contribution
## Your contribution
Yes, I can help by submitting a PR.
I have already prepared a minimal implementation on my fork that:
- adds async tool support to AsyncRolloutWorker
- keeps sync tool behavior unchanged
- preserves current error handling semantics
- adds focused tests
- updates the async GRPO docs accordingly
The scope is intentionally narrow:
- no API change
- no config change
- no attempt to solve partial-rollout resume semantics for in-flight async tools
- no larger agent-loop refactor
The implementation direction is to mirror the mainline GRPOTrainer behavior:
- split tools into sync and async dictionaries
- execute async tools with asyncio.gather(..., return_exceptions=True) within a tool-calling round
- keep result ordering and failure accounting unchanged
If this direction sounds reasonable, I can open a small PR for review.
Contributor guide
Assessment
This issue has not been assessed yet.