deepspeedai / deepspeedai/DeepSpeed
Killing subprocess when saving checkpoints during training with zero2.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Hello,
I've been fine-tuning a qlora of 34B parameter model using the DeepSpeed Zero2 optimization with CPU offloading. The fine-tuning process itself runs without issues. However, when I attempt to save checkpoints, I encounter an error that results in the subprocess being killed. It appears to be caused by running out of CPU memory.
My setup includes a V100 GPU with 32GB of memory, an 8-core CPU, and 32GB of RAM. (the same code works on a similar machine with the same GPU and environment except it has 64G RAM).
Could you suggest any methods to reduce CPU memory usage while saving checkpoints?
Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or entry point is identified in the issue. Start by examining the DeepSpeed ZeRO-2 checkpoint-saving path with CPU offloading and compare the reported 32 GB and 64 GB RAM environments; done would require identifying the memory failure and documenting or implementing a validated reduction in checkpoint-time CPU usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100