NVIDIA-NeMo / NVIDIA-NeMo/RL

generalize metric reporting

Open
#1,434 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
enhancement
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

the metric reporting we have is hard coded at various places which caused us to have this grad norm bug https://github.com/NVIDIA-NeMo/RL/commit/9475e7be77b714954ccc0d3761273b16af0966c3. As an example, the code here in grpo.py https://github.com/NVIDIA-NeMo/RL/blob/9475e7be77b714954ccc0d3761273b16af0966c3/nemo_rl/algorithms/grpo.py#L1143 determines which metric to mean and which to sum, taking the control out of the policy worker's hands. This can cause problems like the grad norm issue. I think a better solution is for the policy worker to tell grpo.py which metric needs to be summed or averaged. Or perhaps average it within the worker itself to avoid the grad norm bug from happening again

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.