Pretraining Throughput — Unexpected Drop in Speed

Hello,

We are currently pretraining the ModernBERT model using the [configuration](https://github.com/AnswerDotAI/ModernBERT/blob/pretraining_documentation/yamls/modernbert/modernbert-base-pretrain.yaml) provided in the `pretraining_documentation` branch, and we're monitoring throughput via Weights & Biases (see the graphs below).

We computed the training speed using the number of steps at various timestamps (e.g., 30, 60, 90, and 120 minutes), and we observed a notable drop in throughput over time. However, the behavior is inconsistent across the wandb metrics:
* The first four metrics (tokens/sec, samples/sec, etc.) show fluctuations but no clear downward trend.
* The last two metrics (`batches_per_sec` and `device/batches_per_sec`) show a consistent decline throughout training.
* Also the speed we acquire does not match with the statistics mentions in your paper.

Please refer to the wandb screenshot below for the graphs.


Questions:
1. What metric(s) shall we consider when computing the total number of training tokens just like in the paper? (We will also use 8xH100 SXM GPUs in DDP setting)
2. Could you clarify the specific meanings of these six throughput metrics?
3. Is it expected to see a continuous drop in `batches_per_sec` over time, while token and sample rates remain relatively stable?
4. Did you observe similar behavior in your runs?
5. What is the reason behind the decline in the last two subfigures?7. Are there known causes (e.g., memory fragmentation, optimizer overhead) that might explain this decline?
6. Would it be possible for you to share your wandb logs with us?

We would really appreciate your insights on whether this behavior is expected or if it's indicative of a bottleneck in our setup.