Drop OLMoE-1B-7B support
- Langage dominant
- Python
- Étoiles
- 1.5k
- Forks
- 315
- Merge moyen
- 1 j 9 h
- PR mergées (30 j)
- 11
Description
Companion to allenai/open-instruct#1837, which drops OLMoE from open-instruct. Filing the OLMo-core half so the two can be coordinated.
## Where OLMoE-1B-7B is supported
- `src/olmo_core/nn/transformer/config.py:1080` — `TransformerConfig.olmoe_1B_7B`
- `src/scripts/train/OLMoE-1B-7B.py` — training script (calls the above at line 26)
- `src/examples/huggingface/convert_checkpoint_from_hf.py:69` — `"olmoe_1b_7b": TransformerConfig.olmoe_1B_7B` in the config lookup
## What should NOT be removed
Several things match a naive `grep -i olmoe` but are unrelated, and removing them would break other models:
- **`DataMix.OLMoE_mix_0824`** (`src/olmo_core/data/mixes/__init__.py:39`) — a data mix, not a model. It is used by `src/scripts/official/OLMo2/OLMo-2-0325-32B-train.py:52` and `src/olmo_core/internal/experiment.py:236`, and covered by `src/test/data/mixes_test.py`.
- **`src/scripts/train/OLMoE2-medium.py` / `OLMoE2-large.py`** — OLMoE2, a different and newer MoE architecture.
- **The MoE machinery itself** (`feed_forward_moe`, the per-expert mappings in `src/olmo_core/nn/hf/convert.py:51-53,149-165`) — shared with OLMoE2 and any future MoE model.
- `OLMoEnvironmentError` and friends in `src/olmo_core/exceptions.py` — substring collision only.
## Background
The prompt for this was a checkpoint-export bug in open-instruct (allenai/open-instruct#1820), where transformers' `conversion_mapping` rewrote weight names on save and produced checkpoints vLLM could not load. OLMoE is the one model in open-instruct's export path that legitimately needs that rewriting, so it is the one case that keeps the export logic branching on model type.
Worth confirming before acting: OLMo-core's converter maps MoE weights to the per-expert HF names and has no fused `gate_up_proj` mapping. If transformers ≥5.5 represents OLMoE with fused expert tensors in memory, the HF conversion path would already fail for OLMoE on that version. If OLMoE-1B-7B is still wanted here, that may be a converter fix rather than a removal.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.