NVIDIA / NVIDIA/Megatron-LM

Add tests for partial cudagraph memory and numerics regressions

Open
#5,594 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Is your feature request related to a problem? Please describe.**
Create tests that can catch regressions for partial cudagraph memory and numerical changes.

- Catching memory regressions is not straightforward since pytorch doesn't provide tooling for tracking the memory usage of graph pools. The current way is to manually comb through the torch memory profile.

- For numerical changes, this is also difficult as pytorch/libraries provide no guarantee cudagraphs call the same kernels as eager mode, meaning without some intervention, graphing over a module will cause numerical changes.

Contributor guide

Open the contributing guide

Research direction

No files or existing tests are named. Start by locating current partial cudagraph and graph-pool coverage, then identify how numerical comparisons are performed between cudagraphs and eager mode. Done means regression tests cover both graph-pool memory behavior and numerical changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.