NVIDIA-NeMo / NVIDIA-NeMo/Gym

Make metrics consume reward_components directly

Open
#2,137 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Problem

BaseMultiRewardVerifyResponse defines reward_components as the canonical multi-reward representation, but aggregate metrics and reward profiling only inspect top-level numeric fields.
Verifiers must therefore duplicate every component:

python
{
    "reward": 2.0,
    "reward_components": {"correctness": 1.0, "format": 1.0},
    "correctness": 1.0,
    "format": 1.0,
}

This creates two sources of truth.

Proposal

Teach generic metrics consumers to read reward_components directly, including:

Aggregate metrics
Reward profiling and rollout_infos
Pass/majority metrics where applicable
Other code that generically extracts numeric verifier fields
Preserve existing output names such as mean/correctness.

During migration, matching duplicated values should be deduplicated; conflicting values should produce a clear error.

Acceptance criteria

  • Per-component metrics work without duplicate top-level fields.
  • Scalar-only verifiers remain unchanged.
  • Existing unrelated top-level numeric fields still work.
  • Missing, inconsistent, and conflicting components have documented behavior.
  • Update the multi-reward example, tests, and documentation to remove the duplication workaround.
  • NeMo RL already consumes reward_components directly for GDPO, so this aligns evaluation with the training contract.

Related: https://github.com/NVIDIA-NeMo/Gym/issues/1664

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Search for generic metric consumers, reward profiling and rollout_infos, pass/majority metrics, and numeric verifier-field extraction. Then inspect the multi-reward example, related tests, and documentation. Done means reward_components works without duplicated fields, scalar verifiers and unrelated numeric fields remain compatible, conflicts have clear behavior, and the workaround is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.