allenai / allenai/open-instruct

GRPO on the olmo-core path bypasses the hybrid weight-loading dispatch

Open
#1,842 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.9k
Forks
585
Avg merge
5d 17h
Merged PRs (30d)
16

Description

Raised by @farhatkevin in review of #1822, and reachable once that PR lands.

`grpo_olmo_core_actor.py` calls olmo-core's `load_hf_model` directly in two places instead of `olmo_core_utils.load_hf_weights_into_olmo_core`, so it never reaches the `olmo_hybrid` dispatch #1822 adds:

- `:136` — the reference policy
- `:206` — the reload after parallelization

This is reachable rather than theoretical: #1822 adds `allenai/Olmo-Hybrid-7B -> olmo3_hybrid_7B` to `OLMO_MODEL_CONFIG_MAP`, and the actor never passes `--config_name`, so `setup_model` resolves the config from `model_name_or_path`. The hybrid olmo-core model builds correctly and only then loads its weights through the wrong path.

It fails loudly, not silently. `_get_converter_from_hf(model_type="olmo_hybrid")` does not raise — it returns the generic llama-style converter — but olmo-core has no `linear_attn` template in either `HF_TO_OLMO_CORE_WEIGHT_MAPPINGS` or `MODEL_TYPE_SPECIFIC_HF_TO_OLMO_CORE_WEIGHT_MAPPINGS` (which covers only `llama`, `qwen3`, `qwen3_5_text`, `gemma3_text`), so every GDN key goes unconsumed and `StateConverter.convert` raises `RuntimeError: Some state keys were not converted: [...]` listing the `model.layers.N.linear_attn.*` keys. Nothing gets written, so there is no risk of training a half-initialized model.

Fixing the dispatch is two lines. Claiming hybrid GRPO support needs more: a run that gets past startup, and a check that generation and the reference policy behave.

Verified by reading the call sites and by introspecting the installed olmo-core converter registry; no GRPO run was attempted.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.