Issue with Recreating RAFT Llama-7b Lora Benchmarks

Hey, our team is trying to recreate the RAFT (RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment) paper HH-RLHF benchmarks with Llama-7b. We successfully did the SFT step, however, when we do the reward modelling, our accuracy is noticeably lower than the one reported in the paper (we're getting ~71% vs the ~79% reported in the paper).
We have about the same training setup:

• Paper: 8x A100 (40GB)
• Our setup: 8x A6000 (48GB)

Also, even with a bit more vram, we can't do batches of 32 (8x4) on the graphics cards because of the reward modelling step requiring both the chosen and rejected pairs being loaded in at the same time, effectively halving the batch size that we can fit. Is this correct?
We followed the RAFT paper closely:

• Used linear lr schedule
• 0.00002 learning rate for SFT
• 0.00003 learning rate for RM
• Same LoRA config (16, 32, .1)
• 32 Batch Size for SFT
• Since we can't fit 32 batch size for RM we tried 16 and 8.
• 1 epoch for both steps

Is use gradient accumulation used? In the paper there was no mention of it, however, we found that we got better accuracies using two steps of gradient accumulation, essentially keeping the batch size at 32.
Any help would be greatly appreciated!

The issue does not name a file, test, or entry point. Start by comparing the reward-modeling batch-size and gradient-accumulation settings with the project's RAFT benchmark configuration and the paper's setup. Done means identifying the cause of the accuracy gap or documenting the correct reproduction settings.