NVIDIA-NeMo / NVIDIA-NeMo/Emerging-Optimizers
Reduce latency
Open
@skyw is already working on this.
Since Feb 24, 2026.
enhancement
- Dominant language
- Python
- Stars
- 274
- Forks
- 51
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 6
Description
Is your feature request related to a problem? Please describe.
CUDA graph: Necessary to reduce the impact of all sorts of latencies.
Multi tensor apply: reduce kernel launches and saves latency. also more bandwidth optimal.
Describe the solution you'd like
All optimizer should support "capturable" argument as native PyTorch does, e.g. https://docs.pytorch.org/docs/stable/generated/torch.optim.adam.Adam_class.html#adam
constants (betas for example), step counter, everything must be on GPU to be CUDA graph capturable.
cc @gdengk @FDecaYed @BoxiangW
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.