modelscope / modelscope/ms-swift
zero3_offload模式下，DataLoader进程由于Fork + COW 机制导致大量内存消耗并OOM
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。
Bug Description / Bug 描述
zero3_offload模式下，DataLoader默认使用fork模式创建进程，此时父进程已offloadd到内存大量数据，由于Fork + COW 机制的原因，在epoch完成后重启DataLoader的时机会消耗大量内并导致OOM。
此issue也提到类似问题 #9649 。
解决办法：
1、开启dataloader_persistent_workers选项能够有一定程度缓解，但还是有大量额外内占用
2、使用spawn模式创建DataLoader进程规避Fork + COW带来的问题（还未验证）
How to Reproduce / 如何复现
ms-swift 版本: 4.3.2
Python: 3.12
触发条件: DeepSpeed ZeRO-3 + torchrun (8 GPU)
模型: Qwen3.6-27B
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the DataLoader construction and the zero3_offload configuration path, then reproduce the reported behavior with Python 3.12, DeepSpeed ZeRO-3, torchrun on 8 GPUs, and the Qwen3.6-27B model. Compare the default fork behavior with persistent workers and the proposed spawn mode. Done means the reproduction no longer causes the reported memory growth and OOM, with coverage for the affected configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100