allenai / allenai/OLMo

Duplicate weight payloads across intermediate checkpoint branches (step101000 / step175000 / step300000)

Aperta
#915 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
type/bug
Lingua principale
Python
Stelle
6.7k
Fork
797
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

### 🐛 Describe the bug

While running probing experiments across OLMo-2-1124-7B intermediate checkpoints, we found that three stage-1 checkpoint branches serve byte-identical model weights despite distinct commits and step labels:

- stage1-step101000-tokens424B (commit 75e0a15f2508b9e98f144362f279698488e79ccc)
- stage1-step175000-tokens735B (commit 990e782724d2f20ebf4bb419c41b7cf933100e60)
- stage1-step300000-tokens1259B (commit b8ddf844c034e3f7e0f85efe613dde7b3c688efd)

All six model-*-of-00006.safetensors shards share identical LFS sha256s across the three branches (first shard: f9acbd5400251483…), verifiable without downloading via the tree API, e.g.:

https://huggingface.co/api/models/allenai/OLMo-2-1124-7B/tree/stage1-step101000-tokens424B

vs the same URL for the other two branch names. Neighboring branches (e.g. stage1-step102000-tokens428B, stage1-step130000-tokens546B, stage1-step301000-tokens1263B) have unique shards, so the issue appears confined to this cluster - though we only checked ~25 branches; a full sweep may find more. We confirmed empirically that the three branches produce bit-identical hidden states.

Possibly related in mechanism to the checkpoint-naming error fixed in allenai/OLMo#883 (1B phase 2), which suggests the publishing pipeline has mislabeled branches before.

Impact: any research using these branches as training-trajectory checkpoints (probing, training-dynamics analyses) silently gets one unknown-step weight set under three labels. Suggested fix: repoint or remove the two mislabeled branches, and possibly publish per-branch shard checksums so users can verify. Happy to share our verification script.

### Versions

Note: the issue is client-independent — the duplicate shard checksums are visible directly in the Hub's HTTP API (/api/models/allenai/OLMo-2-1124-7B/tree/) with no Python involved; versions below are from the environment where we first observed identical model outputs across the branches.

transformers==5.15.0
huggingface-hub==1.27.0
tokenizers==0.22.2
safetensors==0.8.0
datasets==5.0.0
accelerate==1.14.0
torch==2.13.0
numpy==2.4.4

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.