allenai / allenai/open-instruct

numpy_sft cache key is not reproducible across builds, silently discarding tokenization

Aperta
#1,818 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
3.9k
Fork
585
Merge medio
5g 17h
PR unite (30g)
16

Descrizione

## Summary

The `numpy_sft` cache key computed by `olmo_core_finetune.py` is not stable across
builds of this repo. The same tokenization command, with byte-identical cache-key
arguments, produced two different cache directories when run from two images built
from commits that do not touch any hashing code. The training job then reports the
cache as missing and the (16 hour) tokenization has to be redone.

## What happened

Tokenized `allenai/Dolci-Think-SFT` (2.25M rows, 24.7B tokens, ~16 h) with:

```
--model_name_or_path allenai/Olmo-3-1025-7B \
--tokenizer_name_or_path allenai/olmo-3-tokenizer-instruct-dev \
--chat_template_name olmo123 --max_seq_length 32768 \
--mixer_list allenai/Dolci-Think-SFT 1.0 \
--local_cache_dir /weka/oe-adapt-default/allennlp/numpy_sft_cache \
--seed 33333 --cache_dataset_only
```

It wrote `numpy_sft/3f007b1675-dc038a5c`.

The training job, with identical values for every cache-key argument, failed with:

```
FileNotFoundError: Pre-tokenized numpy SFT dataset not found.
expected: .../numpy_sft/bc50d8aba1-dc038a5c
hash: bc50d8aba1
```

Re-running the *tokenization* command verbatim under the newer image also computes
`bc50d8aba1`, so this reproduces and is a property of the build, not of the arguments.

Ruled out: `Dolci-Think-SFT` unchanged since 2026-03-02; `olmo-3-tokenizer-instruct-dev`
unchanged since 2026-02-27; `dataset_transformation.py` and `numpy_dataset_conversion.py`
byte-identical between the two images' base commits. Both images also agree on the
*transformed dataset* hash (`bc50d8aba1`) — only the `numpy_sft` directory name differs,
so the contents are identical and the two can be symlinked.

## Suspected mechanism (not confirmed)

`compute_config_hash` builds `tc_dict` before it first touches `tc.tokenizer`:

```python
tc_dict = {k: v for k, v in asdict(tc).items() if v is not None} # dataset_transformation.py:2060
chat_template = getattr(tc.tokenizer, "chat_template", None) # :2061 <- first access
```

`tokenizer_files_hash` is only assigned as a side effect of the `tokenizer`
`cached_property` (`:904`), and is filtered out of `tc_dict` while it is still `None`.
So whether it contributes to the cache key depends on whether anything accessed
`tc.tokenizer` earlier in the process. It is also annotated `# for tracking purposes`
(`:885`), which suggests it was not meant to be part of the key at all.

I could not confirm this is what differed between our two images, so please treat the
mechanism as a lead rather than a diagnosis.

## Why it matters

The failure is silent and expensive: the only symptom is a cache miss, and the fix
appears to be "tokenize again". For a large mixture that is 7-16 hours per occurrence.

It also means the guidance in `scripts/train/debug/oc_sft_olmo3_7b_1node.sh` — that the
tokenize and train jobs must pass byte-identical cache-key arguments — is necessary but
not sufficient: identical arguments do not guarantee the same key if the image differs.

Related: #1801 fixed a different manifestation (the printed remediation command omitted
`--seed`, `--chat_template_name` and `--transform_fn`, so following it built a cache under
a different key). That change was to the error text only and is not the cause here.

## Suggested direction

Make the key depend only on declared inputs — either exclude `tokenizer_files_hash` from
`tc_dict`, or populate it deterministically before hashing rather than relying on
attribute-access order. Logging the computed hash alongside its inputs on a cache miss
would also make this diagnosable in one run instead of several.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.