Accuracy Investigations (Replicate near-SOTA Baselines)

**Problem Description.**

We need to replicate a systematic near-SOTA baseline from a paper to ensure we're able to launch successful runs from NemoRL. This is an ongoing thing, but we have some initial promise that SkyWorks, a baseline sensitive enough to hyperparameters, is replicable from one of our branches. We want to ensure that is also the case on main.

**Solution Proposal**

Replicate the experiments we've done on our branch on Main, to ensure main also has the capability of training sensitive enough SOTA runs and replicate paper claims.

On our internal branch, we were able to replicate the following curves:

***SkyWorks Run***

Alex was able to run the following run on an internal branch:

The primary curve here is the blue one, which required some tuning of the clipping. It roughly matches the most appropriate baseline from the paper (the tan line which does not use the entropy loss formulation NemoRL does not support).

Link: https://arxiv.org/pdf/2505.22312, Figure 5

This does not validate NemoRL's accuracy, but it gives us additional belief. We would love to replicate this on main.

The config can be found on our lusture path in our NRT clustre:
```/lustre/fs1/portfolios/llmservice/projects/llmservice_modelalignment_sft/users/abukharin/math/launch_scripts/skyworks2.sh```

We will like to have some resources to run the same thing as main, and to ensure that we're able to correctly replicate the same curves from there.

**Alternatives**

SkyWorks is one finalized baseline. We are also running DAPO on VERL (w/ only the features NemoRL supports on main) to set up a comparable baseline with NemoRL from the same branch we tried on SkyWorks. We would like a replication of that on main as well.