pythia-12b: step50000 and step58000 serve the final checkpoint's first shard (embed_in + early layers)

While extracting embedding matrices across the full Pythia checkpoint grid (HTTP range reads only), we found two pythia-12b step branches whose first weight shard is the final model's.

## File-level evidence

`pytorch_model-00001-of-00003.bin` — the shard holding `gpt_neox.embed_in.weight` per `pytorch_model.bin.index.json` on every branch checked — has sha256
`9d21715462588073675eca9c65ba55a47d911da460a6083938d04acd4589f542`
on exactly four refs: `main`, `step143000`, **`step50000`, and `step58000`**. All other 151 step branches have unique first shards, and shards 2–3 (including `embed_out.weight`, shard 3) are unique per branch everywhere. (`main` = `step143000` is expected; the anomaly is the two mid-training branches.)

Hash table attached (`upstream_12b_shard1_hashes.csv`).

## Tensor-level evidence

The input-embedding statistics at steps 50000, 58000, and 143000 are bit-identical, and discontinuous with their neighbours — while the true end of training is smooth (median input-embedding L2 | median cosine-to-initialization):

| step | in-L2 | cos-init |
|---|---|---|
| 49000 | 1.5169 | 0.5935 |
| **50000** | **1.3056** | **0.5110** |
| 51000 | 1.5132 | 0.5869 |
| 57000 | 1.5003 | 0.5695 |
| **58000** | **1.3056** | **0.5110** |
| 59000 | 1.4952 | 0.5646 |
| 141000 | 1.3082 | 0.5112 |
| 142000 | 1.3069 | 0.5111 |
| 143000 | 1.3056 | 0.5110 |

The duplicated values continue the 141000→143000 trend exactly, so the shard's true home is the final checkpoint: `step50000` and `step58000` carry the final model's shard 1 in place of their own.

## Scope

We verified shard 1 by sha256 and by embedding bytes. Shards 2–3 have unique hashes on these branches, so the swap appears limited to shard 1 — but every tensor within shard 1 (embeddings and early layers) on those two branches would be final-model content.

## Impact

Learning-dynamics analyses touching pythia-12b around steps 50000/58000 will silently mix final-model weights into mid-training trajectories. (In our own analyses we drop these two steps and keep the genuine final checkpoint.)

## Reproduction (no downloads required)

1. `GET https://huggingface.co/api/models/EleutherAI/pythia-12b/tree/`; compare `lfs.oid` of `pytorch_model-00001-of-00003.bin` across branches.
2. Range-read `gpt_neox.embed_in.weight` from `step50000` and `step143000` and compare bytes.

Happy to share the extraction code and full per-step statistics.

[upstream_12b_shard1_hashes.csv](https://github.com/user-attachments/files/31050477/upstream_12b_shard1_hashes.csv)

Start by comparing the lfs.oid for pytorch_model-00001-of-00003.bin through the Hugging Face model-tree API, then range-read gpt_neox.embed_in.weight for step50000 and step143000. The reported evidence covers shard 1, but the issue does not specify the repository change or artifact-correction process that would define done.

Written by the indexing model from the issue text.