Questions regarding performance on GSM8kaug

Hi,

I have few questions regarding CoCoNut applied to GSM8k.

1- What is the accuracy after each stage of training?

2- Is the number of generated steps at inference (on the test set) consistent with "ground truth total steps - number of removed steps"? For instance, taking the following example from the test set:

Q: " John drives for 3 hours at a speed of 60 mph and then turns around because he realizes he forgot something very important at home..."
CoT: [
"<<3*60=180>>",
"<<4-2=2>>",
"<<30*.5=15>>",
"<<2-.5=1.5>>",
"<<80*1.5=120>>",
"<<120+15=135>>",
"<<180-135=45>>"
]
A: 45

Is the model necessarily generating 6 steps at stage 1, 5 steps at stage 2 and 0 steps in at stage 3? (by steps I mean explicit ones, after latent reasoning operations)

3- Related to the question above, how do you guarantee that at the final stage the model doesn't output any explicit CoT and that it generates directly the answer after latent thought phase?

4- Did you compute the test accuracy at iteration 0 of stage 1? I'm interested in knowing how much decrease in accuracy is solely due to change in the input sequence.

5- Are \ and \ tokens necessary? Did you test training all stages without them?

6- For experiments with Llama, what is the accuracy with SFT-CoT? Do you also have stage-wise accuracy scores?

Thank you.

The issue names no files, tests, or entry points. Start by reviewing the GSM8kaug training stages and the Llama SFT-CoT experiments, then document the requested stage-wise accuracies, inference-step behavior, final-stage output guarantee, iteration-0 accuracy, token requirements, and Llama results.

Written by the indexing model from the issue text.