CPU offloading in slime is too aggressive
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
It seems that CPU offloading in SGLang utilizes torch_memory_saver and that in Megatron/slime uses cumem_allocator. There is only a global switch that turns on offloading or not, which wastes a massive amount of host memory when GPU memory is sufficient, especially when training small models (e.g. 0.5B-7B). Fine-grained offloading may lower the host memory requirement for training RL models.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the global offloading switch in the Megatron/slime path and comparing its use of cumem_allocator with SGLang's torch_memory_saver. Define which model components should be offloaded selectively, then measure host-memory use on the stated 0.5B–7B training cases while preserving RL training behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100