modelscope / modelscope/ms-swift
训练时，logits.float()新申请的显存数过大导致oom
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
训练参数：local bs=1, max_length=32k, zero3_offload
模型：qwen3-30b-a3b-instruct-2507，
训练方法：继续预训练：
卡：8机*8卡A100
请问logits.float()这个49G哪来？logits.float()这一步应该是把序列的logits从fp16转为fp32，最新需要分配的总显存应该是32k*151936(vocab_size)*4/(1024^3)=18.5G，怎么会有49G呢？
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported configuration: local batch size 1, 32k sequence length, Qwen3-30B-A3B, ZeRO-3 offload, and 8 machines with 8 A100 GPUs each. Compare the memory used by logits.float() with the expected fp32 allocation shown in the issue and inspect the attached profiler screenshots. Done means identifying why the allocation reaches about 49 GiB and documenting or fixing the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100