Rationale for using Llama-3.1-Nemotron-70B-Reward as the Puppeteer state encoder

Thank you for releasing the code for this interesting work!

After reading the Puppeteer implementation, my understanding is that Llama-3.1-Nemotron-70B-Reward-HF is loaded locally and frozen with torch.no_grad(). At each orchestration step, the hidden state of its last token is used as the state representation and passed to an MLP policy network, while only the MLP is optimized using REINFORCE.

I noticed that the README allows users to replace the Reward Model, but I could not find an explanation for why the 70B Nemotron Reward Model was selected as the default state encoder. Could you please clarify the following questions?

What was the main motivation for choosing a 70B Reward Model instead of a smaller language model, reward model, or embedding model?
Is the reward-model-specific representation important for orchestration, or was Nemotron-70B mainly selected as a strong general-purpose state encoder?
Did you conduct any ablation experiments using smaller models, such as 3B, 7B, or 8B encoders? If so, how sensitive is the orchestration performance to the encoder size?

Understanding this design choice would be very helpful for evaluating whether the orchestration capability mainly comes from the reinforcement-trained MLP policy or depends substantially on the representation ability of the 70B model.

Thank you very much for your time!

Start by reading the README's Reward Model configuration and the Puppeteer implementation, then trace how the frozen model's final-token hidden state reaches the MLP policy. Done means documenting the selection rationale and any available encoder-size or ablation evidence, including whether smaller alternatives affect orchestration.