Duplicate weight payloads across intermediate checkpoint branches (step101000 / step175000 / step300000)
- Lingua principale
- Python
- Stelle
- 6.7k
- Fork
- 797
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### 🐛 Describe the bug
While running probing experiments across OLMo-2-1124-7B intermediate checkpoints, we found that three stage-1 checkpoint branches serve byte-identical model weights despite distinct commits and step labels:
- stage1-step101000-tokens424B (commit 75e0a15f2508b9e98f144362f279698488e79ccc)
- stage1-step175000-tokens735B (commit 990e782724d2f20ebf4bb419c41b7cf933100e60)
- stage1-step300000-tokens1259B (commit b8ddf844c034e3f7e0f85efe613dde7b3c688efd)
All six model-*-of-00006.safetensors shards share identical LFS sha256s across the three branches (first shard: f9acbd5400251483…), verifiable without downloading via the tree API, e.g.:
https://huggingface.co/api/models/allenai/OLMo-2-1124-7B/tree/stage1-step101000-tokens424B
vs the same URL for the other two branch names. Neighboring branches (e.g. stage1-step102000-tokens428B, stage1-step130000-tokens546B, stage1-step301000-tokens1263B) have unique shards, so the issue appears confined to this cluster - though we only checked ~25 branches; a full sweep may find more. We confirmed empirically that the three branches produce bit-identical hidden states.
Possibly related in mechanism to the checkpoint-naming error fixed in allenai/OLMo#883 (1B phase 2), which suggests the publishing pipeline has mislabeled branches before.
Impact: any research using these branches as training-trajectory checkpoints (probing, training-dynamics analyses) silently gets one unknown-step weight set under three labels. Suggested fix: repoint or remove the two mislabeled branches, and possibly publish per-branch shard checksums so users can verify. Happy to share our verification script.
### Versions
Note: the issue is client-independent — the duplicate shard checksums are visible directly in the Hub's HTTP API (/api/models/allenai/OLMo-2-1124-7B/tree/) with no Python involved; versions below are from the environment where we first observed identical model outputs across the branches.
transformers==5.15.0
huggingface-hub==1.27.0
tokenizers==0.22.2
safetensors==0.8.0
datasets==5.0.0
accelerate==1.14.0
torch==2.13.0
numpy==2.4.4
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.