Long-Context Data Generation

Goal - Produce the task/answer layer for our long-context extension stage, turning
sourced long documents into training examples that contain real long-range dependencies, for the 11 target EU languages (es fr de it pt pl nl cs ro el uk). This is further described in the writeup.

The examples must force cross-context reasoning, and for low-resource languages that long-range signal has to be synthesized deliberately.

Description

Two candidate generation paths were scoped (writeup §5):

• Path 1 — prompt-driven: a strong multilingual long-context model authors task + answer grounded in the whole document. High diversity/naturalness, but expensive, unverifiable (needs QC/back-translation), and answer quality is bounded by the model's real 120k ability for low-resource langs.
• Path 2 — OLMo bootstrap (longmino_synthetic_cwe_rex): stats pick terms, code computes the answer. This is cheap, fast, verifiable, scales to 120k trivially.

We adopt Path 2 as the backbone but broaden the task layer beyond CWE/REX to address the concern that bootstrap alone limits samples to counting tasks. Concretely, the task mix should also include:

• Multi-document QA — questions whose answers require combining information from ≥2 documents placed in the same context window.
• Long-text summarization — summarize a long single document or a section spanning far into the context.
• Needle-in-a-haystack — insert a fact at a controlled depth and query it; verifiable by construction (we know the needle and its position).

The needle and multi-doc-QA (constructed-answer) variants stay verifiable like CWE; free-form summarization and any model-authored QA go through the Path 1 QC / back-translation pass. tf-idf term extraction needs per-language tokenization (note el / uk are non-Latin).

We can still use path 1 for generating long responses and reasoning traces upto 32K sequence length!

Deliverable scope

Complete when:

• Path 2 (CWE + REX) pipeline ported to multilingual and producing verified examples across all 11 target languages.
• Task mix extended with multi-doc QA, long-text summarization, and needle-in-a-haystack generators, each with a documented target proportion.
• Verifiable tasks (CWE, needle, constructed multi-doc QA) carry programmatic ground-truth checks; non-verifiable tasks (summarization, model-authored QA) have a QC / back-translation review path.
• Examples span the staged curriculum lengths up to ~120k, honoring the 34/66 long/replay mix.
• Output in the agreed training format with retained metadata (language, task type, source doc, needle depth where applicable) for ablations.
• Spot-check report (~50–100 examples/language) covering naturalness, answer correctness, and language fidelity.

Dependencies

Document sourcing (blocking — generation cannot start without it)

To fill a 120k-token window you need ~80–90k words, so sourcing is two problems:
(a) find the longest native documents per language,
(b) construct 120k contexts by concatenating related documents when single ones fall short.

Tier 1 — genuinely long single documents

• EUR-Lex / MultiEURLEX (top pick): legislative/legal texts, individually very long, parallel across all 24 EU official languages — covers every target incl. el, ro. Consolidated codes reach 120k alone. Parallelism lets us build the same task in 11 langs and measure per-language yield.
• Europarl / parliamentary proceedings: long, parallel, formal-conversational register. Weak/absent for uk — pair with Rada (Ukrainian parliament) transcripts + Ukrainian court registry texts.

Tier 2 — bulk web text, filtered for length

• HPLT v2 · CulturaX · FineWeb-2: document-level web crawl with strong per-language coverage incl. uk, el. Sort by doc length per language, keep the top percentile. Bulk volume.

Tier 3 — concatenation material (120k rung)

• Per-language Wikipedia: best substrate for related-doc concatenation (cluster by category/link-graph, concatenate to 120k). Covers all 11 langs.
• The Stack v2 (repo-level): language-agnostic long-range structure; a few % of the mix.

Filtering: apply OLMo's gzip filter (drop most/least-compressible ~20%) to whatever we pick — language-agnostic, ports directly.

Coverage gap — uk sub-task: Ukrainian is not in EUR-Lex/Europarl, the one target the EU-parallel corpora miss. Requires dedicated sourcing (Rada transcripts, uk Wikipedia, HPLT/CulturaX-uk). Track as a sub-task.

Language	Strong sources	Risk
es fr de it pt pl nl	EUR-Lex, Europarl, big Wikipedia, HPLT	fine
cs ro	EUR-Lex/Europarl (parallel) carry it	thinner web tail
el	EUR-Lex parallel + Greek Wikipedia	non-Latin tokenization
uk	not in EUR-Lex/Europarl	biggest gap — Rada / uk-Wiki / HPLT-uk

Other dependencies

• Batch inference infra — vLLM endpoints on Leonardo driven by a batch engine ([synthgen](https://github.com/OpenEuroLLM/synthgen) or Inference Hive). Path 1 / model-authored tasks additionally need a multilingual model (maybe gemma-4) with a genuine 120k input window.
• Per-language tokenizers for tf-idf term extraction (esp. el, uk).