Question about training data mixture ratio and long-context data proportion for reproducing In-Place TTT

Hello In-Place TTT authors,

Thank you very much for releasing the code and for the previous helpful clarifications.

I am trying to reproduce the reported In-Place TTT results, especially the from-scratch 500M/1.5B experiments and the continual pre-training experiments.

I would like to better understand the training data composition, because the data mixture ratio and the proportion of genuine long-context sequences are critical for reproducing the fast-weight dynamics learned by TTT.

The paper mentions that the training data contains:

* general English and Chinese text;
* high knowledge/reasoning density data;
* code data;
* mathematics data;
* multilingual text.

Could you please provide some approximate information about the following aspects?

---

### 1. From-scratch training data mixture

For the 500M and 1.5B experiments, could you provide the approximate ratio of each data category?

For example:

| Data category | Approximate ratio |
| ------------------------------------- | ----------------- |
| General English text | ? |
| General Chinese text | ? |
| High knowledge/reasoning density data | ? |
| Code data | ? |
| Mathematics data | ? |
| Multilingual text | ? |

Even coarse-grained ratios would be very helpful for reproduction.

---

### 2. Proportion of genuine long-context data

Since In-Place TTT relies on sufficiently long continuous contexts to learn meaningful fast-weight updates, could you clarify the proportion of naturally long sequences?

For the 32K context training:

* What percentage of samples are naturally longer than 8K tokens?
* What percentage are longer than 16K tokens?
* What percentage are longer than 32K tokens?

Are the majority of 32K sequences constructed from:

1. naturally long documents (books, papers, repositories, long webpages, etc.), or
2. packing multiple shorter documents?

If packing is used, approximately what fraction of training tokens comes from genuine long documents versus packed short documents?

---

### 3. Public replacement datasets (ProLong / RedPajama)

For community reproduction using public datasets, what mixture would you recommend between long-context data and general pretraining data?

For example:

| Dataset | Approximate ratio |
| -------------------------------------------- | ----------------- |
| ProLong | ? |
| RedPajama / arXiv / general pretraining data | ? |
| Other long-context sources | ? |

More specifically:

* Should ProLong be considered the primary source of long-context sequences?
* Should RedPajama mainly provide general pretraining-style data?
* What fraction of total training tokens should come from long-context documents?

---

### 4. Continual pre-training data mixture

For continual pre-training experiments (Qwen3-4B / LLaMA-3.1-8B), the paper mentions:

* Stage 1: ~20B tokens, 32K context;
* Stage 2: ~15B tokens, 128K context.

Could you provide approximate proportions among:

* short pretraining-style data;
* natural long documents;
* repository-level code;
* retrieval-augmented data;
* synthetic long-context QA data?

Even approximate ratios would significantly improve reproducibility.

---

I completely understand that the original internal datasets may not be publicly released.

However, a high-level but reproducible description of:

1. data mixture ratio;
2. genuine long-context data proportion;
3. recommended public dataset combination (e.g., ProLong + RedPajama);

would already be extremely valuable for the community.

Thank you again for releasing the code and for your previous helpful responses.

The issue asks the authors for training-data mixture ratios, genuine long-context proportions, public dataset recommendations, and continual-pretraining proportions. No files, tests, or implementation entry points are named, so start by reviewing the released code and paper references for any documented dataset details. Done means a maintainer-provided, reproducible description or an explicit statement that the information cannot be shared.

Written by the indexing model from the issue text.