agentscope-ai / agentscope-ai/Trinity-RFT

[BUG] NCCL hang during rollout_weight_sync (ProcessGroup watchdog timeout)

Abierto
#283 1 comentario 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
701
Forks
79
Merge medio
8 h 7 min
PR fusionados (30 d)
1

Descripción

### Bug Description

Please provide a detailed description of the issue you encountered.

### Environment Information

- Python Version: 3.12.4
- GPU: NVIDIA L20-40G * 8
- CUDA Version: 12.4
- Installation Method: git clone
- Trinity-RFT Version: 0.3.0.dev0

### Steps to Reproduce

Please provide a minimal, self-contained, and reproducible example.

1. trinity run --config examples/XXX/XXX.yaml

### Expected Behavior

No interruptions.

### Actual Behavior

During multi-GPU training, the process occasionally crashes with a NCCL watchdog hang. The error happens at rollout_weight_sync and terminates the job with SIGABRT.

### Log Information

```vbnet
ProcessGroupNCCL.cpp:1554 [PG ID 6 PG GUID rollout_weight_sync Rank 3]
ProcessGroup watchdog hang due to timeout
*** SIGABRT received ...
Fatal Python error: Aborted
```

Image

### Question
Is this a known NCCL communication hang issue?
Any recommended configuration or workaround to prevent the crash?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.