Evaluation results differ when `num_prompts_per_step` is different
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
Evaluation results differ when `num_prompts_per_step` is different.
**Steps/Code to reproduce bug**
In the `main` branch, run the two following commands and the results are different:
- `uv run examples/run_eval.py data.dataset_name="HuggingFaceH4/MATH-500" data.dataset_key=test`. This will a score of 76.2. The default value for `num_prompts_per_step` is -1.
- `uv run examples/run_eval.py data.dataset_name="HuggingFaceH4/MATH-500" data.dataset_key=test generation.num_prompts_per_step=128`. This will give a score of 75.6.
Both used greedy decoding.
**Expected behavior**
See above.
**Environment overview (please complete the following information)**
- Environment location: Docker
- Method of install: Building the Release docker container (https://github.com/NVIDIA-NeMo/RL/blob/main/docker/Dockerfile)
- If method of install is [Docker], provide `docker pull` & `docker run` commands used: N/A
**Environment details**
If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version
**Additional context**
N/A
Contributor guide
Assessment
This issue has not been assessed yet.