NVIDIA-NeMo / NVIDIA-NeMo/RL

Unify cache clearing between Dtensor and Megatron backend

Open
#1,888 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

From @yuki-97

1. `clear_cache_every_n_steps` in DTensor: call `torch.cuda.empty_cache()` every n micro batches.
2. `empty_unused_memory_level` in Megatron:
- 0: won't call torch.cuda.empty_cache()
- 1: call gc.collect() and torch.cuda.empty_cache() after use_reference_model; call torch.cuda.empty_cache() before train and after each global train batch
- 2: in addition to 1, call torch.cuda.empty_cache() after optimizer.step()

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.