使用非colocate模式卡死无法继续训练
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
当使用非colocate模式时，训练过程会卡死，GPU利用率为0，log无法继续打印。log停留在
[36m(SGLangEngine pid=413137)[0m
Capturing batches (bs=64 avail_mem=45.06 GB): 61%|██████ | 22/36 [00:01<00:00, 22.88it/s]
Capturing batches (bs=64 avail_mem=45.06 GB): 69%|██████▉ | 25/36 [00:01<00:00, 23.06it/s]
Capturing batches (bs=56 avail_mem=45.06 GB): 69%|██████▉ | 25/36 [00:01<00:00, 23.06it/s]
[36m(SGLangEngine pid=413137)[0m
Capturing batches (bs=48 avail_mem=45.05 GB): 69%|██████▉ | 25/36 [00:01<00:00, 23.06it/s]
Capturing batches (bs=40 avail_mem=45.05 GB): 69%|██████▉ | 25/36 [00:01<00:00, 23.06it/s]
Capturing batches (bs=40 avail_mem=45.05 GB): 78%|███████▊ | 28/36 [00:01<00:00, 23.76it/s]
且nvidia-smi显示
|========================================================|
| 0 N/A N/A 407292 C sglang::scheduler_TP0 0MiB |
| 0 N/A N/A 412618 C ...egatronTrainRayActor.update_weights 0MiB |
| 1 N/A N/A 407293 C sglang::scheduler_TP1 0MiB |
| 1 N/A N/A 413428 C sglang::scheduler 0MiB |
+---------------------------------------------------------------------------------------+
应该是卡死在update_weights步骤了。有大佬知道咋回事吗
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the non-colocate training hang and inspect the update_weights step alongside the captured-batch log output. Compare the scheduler and MegatronTrainRayActor processes shown by nvidia-smi. Done means training continues past batch capturing, logs resume, and GPU utilization no longer remains at 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100