modelscope / modelscope/ms-swift
我用megtron sft训练了qwen2.5 vl 32b，保存了参数、优化器状态、rng。训练断了，我想在断点恢复训练，但是加载爆显存。不加载优化器和rng可行，但是这样就不符合断点重启的需求了，也不想用optimizer_cpu_offload，会慢。有啥办法吗？
Open
Nobody has claimed this yet.
bug
stale
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
This issue has no description.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or entry points are mentioned. Start by reproducing checkpoint resume for the Megatron SFT Qwen2.5-VL 32B case with optimizer and RNG state loading enabled, then compare it with loading model parameters alone and with optimizer CPU offload. Done means full checkpoint recovery works without exceeding GPU memory and without relying on the slower offload option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100