huggingface / huggingface/trl

AsyncGRPO: support async tool calls in AsyncRolloutWorker

Open
#5,444 2 comments 0 reactions 1 assignee Claimed by @AmineDiro View on GitHub
Dominant language
Python
Stars
19.3k
Forks
3k
Avg merge
1d 20h
Merged PRs (30d)
194

Description

### Feature request

## Feature request

GRPOTrainer already supports async tools, but trl.experimental.async_grpo.AsyncGRPOTrainer currently does not.

At the moment, AsyncRolloutWorker rejects coroutine tools during initialization with:

ValueError("Asynchronous tools are not supported in AsyncRolloutWorker yet.")

This creates a feature gap between the mainline and experimental GRPO implementations.

I would like to propose adding async tool support to trl.experimental.async_grpo.async_rollout_worker.AsyncRolloutWorker with a minimal parity patch.

Relevant code:

- Mainline async tool support in GRPOTrainer
- Experimental async trainer in trl.experimental.async_grpo.AsyncGRPOTrainer

Related merged PR:

- PR #4742 added async tool calls to mainline GRPOTrainer

No paper is directly required for this proposal since this is a feature-parity / implementation-gap request rather than a new algorithm proposal.

### Motivation

## Motivation

Many agent tools are naturally I/O-bound, for example:

- external APIs
- retrieval
- browser / sandbox environments
- RPC-backed tools

In these cases, forcing tools to be synchronous serializes a latency-heavy part of the rollout pipeline.

Since GRPOTrainer already supports async tools, it is surprising that AsyncGRPOTrainer still rejects them. This makes it harder to use AsyncGRPOTrainer for realistic
agent training settings, even though it already supports:

- async reward functions
- tool use
- environment_factory

So the motivation is mainly:

- close the feature gap with mainline GRPOTrainer
- make async agent environments usable in AsyncGRPOTrainer
- do so with minimal scope and without changing the public API

### Your contribution

## Your contribution

Yes, I can help by submitting a PR.

I have already prepared a minimal implementation on my fork that:

- adds async tool support to AsyncRolloutWorker
- keeps sync tool behavior unchanged
- preserves current error handling semantics
- adds focused tests
- updates the async GRPO docs accordingly

The scope is intentionally narrow:

- no API change
- no config change
- no attempt to solve partial-rollout resume semantics for in-flight async tools
- no larger agent-loop refactor

The implementation direction is to mirror the mainline GRPOTrainer behavior:

- split tools into sync and async dictionaries
- execute async tools with asyncio.gather(..., return_exceptions=True) within a tool-calling round
- keep result ordering and failure accounting unchanged

If this direction sounds reasonable, I can open a small PR for review.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.