NVIDIA-NeMo / NVIDIA-NeMo/RL

Evaluation results differ when `num_prompts_per_step` is different

Open
#563 1 comment 0 reactions 1 assignee Claimed by @yuki-97 View on GitHub
bug external x-google
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

Evaluation results differ when `num_prompts_per_step` is different.

**Steps/Code to reproduce bug**

In the `main` branch, run the two following commands and the results are different:

- `uv run examples/run_eval.py data.dataset_name="HuggingFaceH4/MATH-500" data.dataset_key=test`. This will a score of 76.2. The default value for `num_prompts_per_step` is -1.
- `uv run examples/run_eval.py data.dataset_name="HuggingFaceH4/MATH-500" data.dataset_key=test generation.num_prompts_per_step=128`. This will give a score of 75.6.

Both used greedy decoding.

**Expected behavior**

See above.

**Environment overview (please complete the following information)**

- Environment location: Docker
- Method of install: Building the Release docker container (https://github.com/NVIDIA-NeMo/RL/blob/main/docker/Dockerfile)
- If method of install is [Docker], provide `docker pull` & `docker run` commands used: N/A

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**
N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.