NVIDIA-NeMo / NVIDIA-NeMo/RL

Megatron has left-over memory after refit completed

Open
#1,739 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

Ideally the memory should be exactly equal to 0 after refit completed, but megatron will have some left.
```
> assert current_allocated == 0.0, "Memory should be 0 after refit completed"
E AssertionError: Memory should be 0 after refit completed
E assert 0.00244140625 == 0.0
```

Repro:
Set `train_backend == megatron` in the below unit test and run it.
https://github.com/NVIDIA-NeMo/RL/blob/ba46741f081b6a71a68af1d884c71f65b4da80f4/tests/unit/models/generation/test_vllm_generation.py#L1667-L1669.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.