Large Variability Across Seeded Experiments with CTC Loss

Hi all,

I am currently running experiments using the MLSUPERB setup in ESPnet (based on the 202511 version of the codebase), and I would like to ask about the level of variability I am observing across runs, even when I control for randomness using the same seed. The code for my implementation can be found here: https://github.com/josecruzado21/espnet_ctc_dro/tree/ml-superb-experiments

My setup is very close to the default pipeline, with only minor modifications to restrict training to a smaller subset of languages and datasets. The configuration is as follows:

- Data: Subset of MLSUPERB2
- Languages and datasets: English (VoxForge), German (VoxForge), Hebrew (FLEURS), Japanese (FLEURS), Russian (FLEURS), Spanish (FLEURS)
- 1 hour of training data per language
- Pretrained model: MMS 300M (facebook/mms-300m)
- Full fine-tuning of all pretrained weights
- Additional attention head on top of the encoder embeddings for ASR
- CTC-only training (ctc_weight = 1, using ESPnet’s built-in CTC)
- Training for 40 epochs
- Batch size = 4 with gradient accumulation over 16 steps

To control randomness, I set seed=0 in ESPnet. I also verified that batching, sorting, and the initial forward passes are deterministic: the per-sample CTC losses are identical across runs up until the first backward pass (i.e., batch 17 in epoch 1).

However, I observe significant variability in final performance across runs. In the case of Japanese, the difference between the best and worst CER over 5 runs is approximately 19 percentage points. I am attaching a plot showing the CERs for the five runs, along with tables summarizing the results.

I understand that CTC loss may exhibit non-deterministic behavior in the backward pass (depending on the implementation and hardware), but I would like to ask:

- Is this level of variability expected in this setup?
- Are there known sources of non-determinism in ESPnet (or PyTorch CTC) that could explain differences of this magnitude?
- Are there recommended practices to reduce this variability (e.g., specific flags, deterministic settings, or alternative implementations)?

Any guidance would be greatly appreciated