Vision CP crashes when a packed sample cannot populate every CP rank
- 主要語言
- Python
- 星號
- 726
- 分支
- 48
- PR 合併指標
- 30 天內沒有已合併 PR
描述
### Reproduction
- Molmo2-4B SFT checkpoint on 8 A100s with CP=8
- `model.apply_cp_to_vision_backbone=true`
- Debug mixture, dynamic packing, 36,864-token context, device batch size 1
### Actual behavior
On the first training step, ranks 5-7 receive image boundaries with identical
start and end indices. The vision backbone passes the resulting zero-image tensor
to SDPA, which fails with:
```text
RuntimeError: batch size must be positive
```
The remaining ranks wait in NCCL until the watchdog terminates the job.
**`compute_image_shard_boundaries()` tries to move whole image/frame boundary units
between ranks, but it cannot make every shard nonempty when there are fewer usable
units than CP ranks.**
Could you kindly provide some help and guidance on this? Thank you! @chrisc36 @sangho-vision @zixianm
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
Start at compute_image_shard_boundaries() and trace how its image or frame boundary output is consumed by the vision backbone across CP ranks. Reproduce the Molmo2-4B SFT setup from the issue, then verify that ranks do not pass zero-image tensors to SDPA or remain blocked in NCCL when usable units are fewer than CP ranks.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- distributed-systems, machine-learning
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100