allenai / allenai/molmo2

Vision CP crashes when a packed sample cannot populate every CP rank

未關閉
#26 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
726
分支
48
PR 合併指標
30 天內沒有已合併 PR

描述

### Reproduction

- Molmo2-4B SFT checkpoint on 8 A100s with CP=8
- `model.apply_cp_to_vision_backbone=true`
- Debug mixture, dynamic packing, 36,864-token context, device batch size 1

### Actual behavior

On the first training step, ranks 5-7 receive image boundaries with identical
start and end indices. The vision backbone passes the resulting zero-image tensor
to SDPA, which fails with:

```text
RuntimeError: batch size must be positive
```

The remaining ranks wait in NCCL until the watchdog terminates the job.
**`compute_image_shard_boundaries()` tries to move whole image/frame boundary units
between ranks, but it cannot make every shard nonempty when there are fewer usable
units than CP ranks.**

Could you kindly provide some help and guidance on this? Thank you! @chrisc36 @sangho-vision @zixianm

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

Start at compute_image_shard_boundaries() and trace how its image or frame boundary output is consumed by the vision backbone across CP ranks. Reproduce the Molmo2-4B SFT setup from the issue, then verify that ranks do not pass zero-image tensors to SDPA or remain blocked in NCCL when usable units are fewer than CP ranks.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
distributed-systems, machine-learning
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。