NVIDIA-NeMo / NVIDIA-NeMo/RL

More robust log prob error checking in nightly tests

Open
#1,039 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

Currently we check log prob error like this:
```
uv run tests/check_metrics.py $JSON_METRICS \
'mean(data["train/token_mult_prob_error"]) < 1.1'
```

But we may sometimes have outliers that skew it way above 1.1. This seems to be more of an issue with very long context generations.

One idea may be to add support for:
```py
# exclude the top 5% of values
mean_exclude_percentile(data["train/token_mult_prob_error"],top=0.05) < 1.1
```
which would be more resistant to outliers. This wouldn't solve the issue for nightly tests without 1-5 steps since the variance may be too large there. In that case we should maybe exit with an agreed upon exit code that can be interpreted as "warning".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.