[ARCH-PROP] Next Concept Prediction

### Architecture Name

Next Concept Prediction

### Parent issue

#10

### Motivations

The success of generative pretraining has largely been driven by predicting future units. However, conventional language modeling remains centered on Next Token Prediction (NTP), where the prediction target is a single token.

Several successful learning paradigms have moved their prediction targets beyond low-level observations. Instead of predicting individual pixels, actions or tokens, they predict latent representations, trajectories or other higher-level units. Predicting multiple future tokens can make the objective more difficult, but the prediction still remains in the token space.

Next Concept Prediction (NCP) introduces a prediction objective in a discrete latent space. A concept spans multiple tokens and represents a higher-level unit than an individual token. Predicting the next concept therefore requires the model to infer longer-range structure and provides a harder pretraining task than predicting the next token alone.

### Proposed Architecture

Our model follows the ConceptLM three-module structure:
Token-level Encoder
-> Concept-level Module (with a quantizer to discretize the hidden representation)
-> Token-level Decoder
-> Next-token prediction

The total training objective is:
L_total = L_NTP
+ lambda_vq * L_VQ
+ lambda_ncp * L_NCP
+ lambda_route * L_route

### Preliminary Results (if any)

Verified the architecture on GPT-2, Pythia, and Olmo-3 (6T tokens).

### Experiments Plan

For each model-size pair, the baseline and proposed model will use the same tokenizer, training data and order, token budget, sequence length, optimizer, learning-rate schedule, batch size, random seed, distributed topology and evaluation protocol.

The current 7B reference setting uses sequence length 8192, global batch size 512, concept chunk size 4, Dolma3 training data and approximately 6T training tokens.

The experiments will include the following ablations:

Vanilla OLMo 3 baseline.
Continuous Concept OLMo without VQ.
Concept OLMo with Product VQ but without DD or cross-module routes.
Concept OLMo with VQ and self-DD only.
Full Concept OLMo with VQ, self-DD, encoder-read routes, decoder-read routes and final concept routing.
Merge-mode comparisons between raw_logits, softmax and hard_top1.
Evaluation will include training loss, validation perplexity, matched-token downstream results, HLM prediction quality, VQ reconstruction quality, codebook entropy, dead-code ratio, maximum code usage share, route-weight entropy, peak memory, training tokens per second, MFU and total training time.