Knowledge transfer from pre-pretraining

💡 The Name

Would pre-pretraining with artificial languages like Dyck or Shuffle Dyck be beneficial when transfering to other natural languages other than English, and further, multilingual?

🧠 The "Why" (Problem & Motivation)

Hu et al., (2025) and Shinnick et al., (2025) (and more papers actually, just that these two are quite recent) show that pre-pretraining a LLM with artificial languages like Dyck and Shuffle Dyck helps saving tokens in subsequent training with English. That is, a smaller number of tokens is needed to reach a certain loss value, compared to without pre-pretraining. Another benefit would be to help with reasoning capability. They suggest that pre-pretraining can be a reliable weight initialization method.

However, there are a few gaps (which should be interesting to us I think):

1. Previous experiments were only with English, but not with other languages, let alone multilingual.
2. In these experiments, the models are trained with a relatively small number of training tokens (very much less than Chinchilla-optimal), so the behavior could have been noises in the undertrained regime.
3. The models are trained with only one random seed run, so it might have been just luck and/or noises.
4. Bracket representation and tokenizer behavior: the two above papers represent brackets as non-negative integers (less than 3 digits), and they use existing tokenizers like GPT-NeoX or GPT-2. These two tokenizers tokenize numbers as groups of 3 digits, so these combinations worked out for them. However, other tokenizers like Gemma 3 tokenize numbers digit-by-digit, which will break the intended Dyck signal. => There's no guarantee that this could be generalized to other tokenizers.
5. Relatively "old" architectures as well: their models were based on Pythia (GPT-NeoX) and GPT-2, respectively, which were released a few years ago already => There's no guarantee that this could be generalized to more recent architectures as well (dense only, not sure about MoE).

🚀 The Big Idea

A very simple idea: expand the approach to other languages and multilingual to see if it actually works or not, or the claim can only be limited to English only. For example, Shuffle Dyck seems to resemble the cross-serial dependency feature in some languages, say Dutch, more than English. Let's say that if it actually works for English, then pre-pretraining with Shuffle Dyck should be even more beneficial to Dutch, right?

A beneficial result would be showing that the procedure can translate into better downstream performance rather than just language modeling.

Ideally, we should do the experiments with multiple (at least 3) random seed runs.

🎯 The Dream Outcome

If it actually works, great. Then maybe we can apply this result to model training.
But if not, then we need to warn the community, not against the idea, but to be more careful.

⚙️ Prototype (Optional)

I already got some working code, just need to scale it up :)

Basically I'm planning to do experiments with Llama architecture with Llama tokenizer, in two sizes: 160M and 1B.

❓ Things I'm Figuring Out (Optional)

Right now I'm not sure if I should also experiment with other tokenizers, say Gemma, as well, because then the number of experiments will surely explode.

📎 References and Similar Works

Michael Y. Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. 2025. Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9691–9709, Vienna, Austria. Association for Computational Linguistics.

Shinnick, Z., Jiang, L., Saratchandran, H., Hengel, A.V., & Teney, D. (2025). Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning. ArXiv, abs/2505.22308.

💬 Discussion Notes

Drop the TL;DR of decisions, pivots, and feedback here as people weigh in.

Start from the existing prototype, whose file or entry point is not specified, and establish how it runs the planned Llama experiments at 160M and 1B. Compare pre-pretraining across non-English and multilingual settings, using multiple random seeds where feasible, and evaluate downstream performance as well as language-model loss. Done means a documented set of results showing whether the claimed transfer generalizes.