Ray OOM when saving checkpoint unless both --no-save-optim and --disable-weights-backuper are enabled
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
Your Question
Hi, thanks for the great work.
I encountered a Ray OOM issue when saving checkpoints during training. The training can run normally if I add both of the following arguments:
--no-save-optim --disable-weights-backuper
However, both flags are required. If either one is missing, Ray OOM still happens during checkpoint saving.
The problem is that with --no-save-optim, the optimizer state is not saved, so I cannot properly resume training from checkpoints. I would like to know whether this is a known issue and what the recommended solution is.
Error log
(raylet, ip=10.22.72.139) [2026-04-17 15:06:03,782 E 622 622] (raylet) node_manager.cc:3250:
1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node
Refer to the documentation on how to address the out of memory issue:
https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html
Have you encountered this issue before?
What is the recommended way to avoid Ray OOM during checkpoint saving while still keeping the checkpoint resumable, including optimizer states?
Is there a suggested configuration for saving optimizer states or disabling the weights backuper without losing resume capability?
What I've Tried
--no-save-optim --disable-weights-backuper
Environment (if relevant)
- slime version: 0.2.4
- Python version: python3.12
Additional Context
No response
Pre-submission Checklist
- I have read the CONTRIBUTING.md and understand the collaboration scope.
- I have read the documentation and FAQ and my question is not answered there.
- I have searched for existing issues and my question has not been asked before.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source file or test; start by reproducing checkpoint saving in slime 0.2.4 with optimizer-state saving enabled and compare the four combinations of --no-save-optim and --disable-weights-backuper. Read the linked Ray OOM-prevention guidance, and consider the issue resolved only when a verified configuration preserves resumable checkpoints with optimizer states.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100