CPU RAM leakage as training progresses
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
Following the scripts to run SFT and RL on Qwen3-4B, both training runs resulted in slow memory leakage (see Fig 1 for RL screenshot and Fig 2 for SFT screenshot).
While this is ok for smaller models, it can result in out of RAM crashes for larger models and longer training runs.
Fig 3 shows a case when I was finetuning GLM 4.5 air with 4 H200 nodes, and after 12 hours of training we ran out of memory.
Here's the error logs: https://gist.github.com/zkx06111/5ba1106d5086feea3759ceb24f46c48e
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SFT and RL training scripts referenced in the report, then inspect the linked error logs while monitoring CPU RAM usage over time. Done means the progressive memory growth is identified and resolved so extended training runs do not end in out-of-memory crashes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100