Hub think-dev tokenizer breaks SFT label derivation; #1879 does not reach it

Claude Code · session 6b836cd5 (claude.ai/code), for @abhishekraok

## Summary

`allenai/olmo-3.2-tokenizer-think-dev` cannot be used to tokenize SFT data (every row is dropped) and silently mislabels DPO rows (`` masked out of the last turn). generation-blocks-sft-labels (#1879) does **not** change this: it adds `{% generation %}` blocks to the in-repo `CHAT_TEMPLATES` only, and hub tokenizers still take the prefix-derivation path. The `docs/olmo3.md` workaround (SFT with `instruct-dev`) is correct and lossless, not a stopgap, but the DPO recommendation in the same doc is affected.

## The two tokenizers differ by seven characters

`instruct-dev` and `think-dev` share one template body. The only diff is in the `add_generation_prompt` branch:

```
'<|im_start|>assistant\n' -> '<|im_start|>assistant\n'
```

That branch is off during SFT tokenization, so the two templates render byte-identical SFT text. `think-dev` only changes inference/eval/RL prompts (force-think). The `*-release` repos add the Olmo identity system prompt on top; same body otherwise.

## Measured

Two-turn think conversation (`T1A1`, `T2A2`) through `_tokenize_tulu_sft_with_assistant_labels`, transformers 5.4.0, identical results on `main` and on the #1879 head:

| tokenizer | SFT (`last_turn_only=False`) | DPO (`last_turn_only=True`) |
|---|---|---|
| `olmo-3-tokenizer-instruct-dev` | correct, `` trained | correct |
| `olmo-3.2-tokenizer-think-dev` | **row dropped** (`Assistant label span ... does not cover its content`) | `'THINKTWOANSWERTWO<|endoftext|>'` — **`` masked, row kept** |

Cause: the derivation locates the start of an assistant turn by rendering the prefix *with* `add_generation_prompt=True` so the header stays masked. With `think-dev` that header ends in ``, which swallows the content's own ``. This is the "think token chopping" bug `docs/olmo3.md` describes, and it lives in the derivation, not in the template.

Impact today: `docs/olmo3.md` recommends `think-dev` for DPO. DPO rows tokenized that way exclude `` from both chosen and rejected logprobs. Small effect, but unintended and unlogged.

## Proposed fix

Make the prefix derivation robust to a generation prompt that ends in content the assistant will also produce. Either:

- derive the header end from the prefix rendered **without** the generation prompt plus the fixed header string, or
- after locating the header, back the span start up over any header suffix that is also a prefix of the message content (`` here).

Either is a few lines in `_tokenize_tulu_sft_with_assistant_labels`, applies to any hub template, and lets SFT, DPO, and eval share one tokenizer. Add `think-dev` (or a fixture with `` in the generation prompt) to `TestChatTemplateAssistantLabelSweep`; nothing in the sweep exercises this today.

Then update `docs/olmo3.md`: say `instruct-dev` and `think-dev` render identically for SFT so either is fine, and drop the "we plan to fix the masking bug" caveat once this lands.

## Scope decision for @farhatkevin

RL is being deprecated in this repo in favor of miles, so only SFT and DPO matter. Of the scripts that pass an in-repo thinker template (`olmo_thinker`, `tulu_thinker*`), exactly one is SFT/DPO: `scripts/train/olmo3/7b_think_dpo.sh`, the historical Olmo 3 recipe. All others are RL. Olmo 3.2+ recipes use the hub tokenizers.

So thinker-templates-undecidable (#1804) and #1879 fix a path that current SFT/DPO does not use. Options:

1. Shelve #1879, close #1804 as superseded by this issue, and fix the derivation here instead.
2. Land #1879 anyway (with the `add_bos` gate fix from the review) if the in-repo thinker templates are staying.

Kevin's call. Either way this issue stands on its own: it is about hub tokenizers, which #1879 does not touch.

Longer term: collapse the four tokenizer repos into one template with `enable_thinking` and identity flags (Qwen3 style). Needs `chat_template_kwargs` plumbing through vLLM and oe-eval, so out of scope here.

Refs #1800, #1804, #1879.