allenai / allenai/open-instruct
GRPO on the olmo-core path bypasses the hybrid weight-loading dispatch
- Dominant language
- Python
- Stars
- 3.9k
- Forks
- 585
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 16
Description
Raised by @farhatkevin in review of #1822, and reachable once that PR lands.
`grpo_olmo_core_actor.py` calls olmo-core's `load_hf_model` directly in two places instead of `olmo_core_utils.load_hf_weights_into_olmo_core`, so it never reaches the `olmo_hybrid` dispatch #1822 adds:
- `:136` — the reference policy
- `:206` — the reload after parallelization
This is reachable rather than theoretical: #1822 adds `allenai/Olmo-Hybrid-7B -> olmo3_hybrid_7B` to `OLMO_MODEL_CONFIG_MAP`, and the actor never passes `--config_name`, so `setup_model` resolves the config from `model_name_or_path`. The hybrid olmo-core model builds correctly and only then loads its weights through the wrong path.
It fails loudly, not silently. `_get_converter_from_hf(model_type="olmo_hybrid")` does not raise — it returns the generic llama-style converter — but olmo-core has no `linear_attn` template in either `HF_TO_OLMO_CORE_WEIGHT_MAPPINGS` or `MODEL_TYPE_SPECIFIC_HF_TO_OLMO_CORE_WEIGHT_MAPPINGS` (which covers only `llama`, `qwen3`, `qwen3_5_text`, `gemma3_text`), so every GDN key goes unconsumed and `StateConverter.convert` raises `RuntimeError: Some state keys were not converted: [...]` listing the `model.layers.N.linear_attn.*` keys. Nothing gets written, so there is no risk of training a half-initialized model.
Fixing the dispatch is two lines. Claiming hybrid GRPO support needs more: a run that gets past startup, and a check that generation and the reference policy behave.
Verified by reading the call sites and by introspecting the installed olmo-core converter registry; no GRPO run was attempted.
Contributor guide
Assessment
This issue has not been assessed yet.