agentscope-ai / agentscope-ai/Trinity-RFT

[BUG] NCCL hang during rollout_weight_sync (ProcessGroup watchdog timeout)

Open
#283 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
701
Forks
79
Avg merge
8h 7m
Merged PRs (30d)
1

Description

### Bug Description

Please provide a detailed description of the issue you encountered.

### Environment Information

- Python Version: 3.12.4
- GPU: NVIDIA L20-40G * 8
- CUDA Version: 12.4
- Installation Method: git clone
- Trinity-RFT Version: 0.3.0.dev0

### Steps to Reproduce

Please provide a minimal, self-contained, and reproducible example.

1. trinity run --config examples/XXX/XXX.yaml

### Expected Behavior

No interruptions.

### Actual Behavior

During multi-GPU training, the process occasionally crashes with a NCCL watchdog hang. The error happens at rollout_weight_sync and terminates the job with SIGABRT.

### Log Information

```vbnet
ProcessGroupNCCL.cpp:1554 [PG ID 6 PG GUID rollout_weight_sync Rank 3]
ProcessGroup watchdog hang due to timeout
*** SIGABRT received ...
Fatal Python error: Aborted
```

Image

### Question
Is this a known NCCL communication hang issue?
Any recommended configuration or workaround to prevent the crash?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.