[MolmoE-1B] ValueError: Unrecognized model in allenai/OLMoE-1B-7B-0924

Hello there, thank you so much for open sourcing Molmo.

I was trying to run inference for MolmoE-1B model using the command `torchrun --nproc-per-node 2 launch_scripts/eval_downstream.py MolmoE-1B-0924 text_vqa --save_dir new_dir` and keep running into `ValueError: Unrecognized model in allenai/OLMoE-1B-7B-0924`. I don't think this is an accurate error message since `OLMoE-1B-7B-0924` contains the OLMoE string which is one of the recognized model types as indicated in the full stack trace below.

**Eval using the same command works for Molmo-7B-D model in the same conda environment.**
Here is the version information for my conda environment:
```
triton==2.3.1
torch==2.3.1
torchmetrics==1.7.0
torchvision==0.18.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
megablocks @ git+https://github.com/Muennighoff/megablocks.git@4a25bc7b5665bcb9da93d72d5ad0c14d41e1a351
```

Full stack trace of the error:

```
torchrun --nproc-per-node 1 launch_scripts/eval_downstream.py MolmoE-1B-0924 text_vqa --save_dir new_dir

[2025-04-01 17:44:33] INFO [root, rank=0] args.save_dir='new_dir'
[2025-04-01 17:44:33] INFO [scripts.mm_eval:142, rank=0] Loading model without FSDP...
[2025-04-01 17:46:13] INFO [olmo.model:2402, rank=0] Checkpoint weight dtype: torch.float32
[2025-04-01 17:46:23] INFO [scripts.mm_eval:203, rank=0] Model weight dtype: torch.float32
[2025-04-01 17:46:23] INFO [scripts.mm_eval:204, rank=0] Total number of parameters: 7,221,875,712
[2025-04-01 17:46:23] INFO [scripts.mm_eval:205, rank=0] Number of non-embedding parameters: 7,118,590,976
[2025-04-01 17:46:24] INFO [scripts.mm_eval:206, rank=0] Peak GPU Memory (MB) before FSDP: 28889
[2025-04-01 17:46:24] INFO [root, rank=0] Starting inference text_vqa
[2025-04-01 17:46:24] CRITICAL [olmo.util:174, rank=0] Uncaught ValueError: Unrecognized model in allenai/OLMoE-1B-7B-0924. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: albert, align, altclip, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, colpali, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deformable_detr, deit, depth_anything, depth_pro, deta, detr, diffllama, dinat, dinov2, dinov2_with_registers, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, emu3, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, git, glm, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hiera, hubert, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mistral3, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_vl, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, seamless_m4t, seamless_m4t_v2, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip_vision_model, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, table-transformer, tapas, textnet, time_series_transformer, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xmod, yolos, yoso, zamba, zamba2, zoedepth
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /private/home/yashs/projects/molmo-experiments/launch_scripts/eval_downstream.py:236 in │
│ │
│ 233 │
│ 234 │
│ 235 if __name__ == "__main__": │
│ ❱ 236 │ main() │
│ 237 │
│ │
│ /private/home/yashs/projects/molmo-experiments/launch_scripts/eval_downstream.py:232 in main │
│ │
│ 229 │ │ overrides = [clean_opt(arg) for arg in other_args] │
│ 230 │ │ config = OmegaConf.merge(config, OmegaConf.from_dotlist(overrides)) │
│ 231 │ │ cfg = cast(EvalConfig, OmegaConf.to_object(config)) │
│ ❱ 232 │ ModelEvaluator(cfg).run() │
│ 233 │
│ 234 │
│ 235 if __name__ == "__main__": │
│ │
│ /private/home/yashs/projects/molmo-experiments/scripts/mm_eval.py:267 in run │
│ │
│ 264 │ │ │ │ max_steps = None │
│ 265 │ │ │ │
│ 266 │ │ │ if evaluation.data.multi_modal == "torch": │
│ ❱ 267 │ │ │ │ dataloader = build_torch_mm_eval_dataloader( │
│ 268 │ │ │ │ │ device_batch_size, │
│ 269 │ │ │ │ │ config.seed, │
│ 270 │ │ │ │ │ model.config, │
│ │
│ /private/home/yashs/projects/molmo-experiments/olmo/data/__init__.py:75 in build_torch_mm_eval_d │
│ │
│ 72 def build_torch_mm_eval_dataloader( │
│ 73 │ batch_size, seed, model_config, data_config, pad_batches, max_steps=None │
│ 74 ): │
│ ❱ 75 │ preprocessor = build_mm_preprocessor( │
│ 76 │ │ model_config, for_inference=data_config.for_inference, shuffle_messages=data_confi │
│ 77 │ │ require_image_features=pad_batches │
│ 78 │ ) │
│ │
│ /private/home/yashs/projects/molmo-experiments/olmo/data/__init__.py:48 in build_mm_preprocessor │
│ │
│ 45 │ │ │ default_inference_len=model_config.default_inference_len │
│ 46 │ │ ), │
│ 47 │ │ MultiModalPreprocessor( │
│ ❱ 48 │ │ │ tokenizer=model_config.get_tokenizer(), │
│ 49 │ │ │ normalize=str(v_cfg.image_model_type), │
│ 50 │ │ │ crop_mode=model_config.crop_mode, │
│ 51 │ │ │ max_crops=model_config.max_crops, │
│ │
│ /private/home/yashs/projects/molmo-experiments/olmo/config.py:770 in get_tokenizer │
│ │
│ 767 │ def get_tokenizer(self): │
│ 768 │ │ tokenizer_cfg = self.tokenizer │
│ 769 │ │ kargs = {} │
│ ❱ 770 │ │ return build_tokenizer( │
│ 771 │ │ │ tokenizer_cfg.identifier, │
│ 772 │ │ │ tokenizer_dir=tokenizer_cfg.tokenizer_dir, │
│ 773 │ │ │ pad_tokenizer_to=self.vocab_size if self.pad_tokenizer else None, │
│ │
│ /private/home/yashs/projects/molmo-experiments/olmo/tokenizer.py:84 in build_tokenizer │
│ │
│ 81 │ # Stop multiple processes on one node trying to download and cache the tokenizer │
│ 82 │ # files, which seems to rarely cause an error │
│ 83 │ if get_local_rank() == 0: │
│ ❱ 84 │ │ tokenizer = AutoTokenizer.from_pretrained( │
│ 85 │ │ │ tokenizer_type, │
│ 86 │ │ │ token=environ.get("HF_ACCESS_TOKEN"), │
│ 87 │ │ │ cache_dir=cache_dir, │
│ │
│ /private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/transformers/models/auto/tok │
│ │
│ 927 │ │ │ │ │ config_dict = load_gguf_checkpoint(gguf_path, return_tensors=False)[" │
│ 928 │ │ │ │ │ config = AutoConfig.for_model(**config_dict) │
│ 929 │ │ │ │ else: │
│ ❱ 930 │ │ │ │ │ config = AutoConfig.from_pretrained( │
│ 931 │ │ │ │ │ │ pretrained_model_name_or_path, trust_remote_code=trust_remote_cod │
│ 932 │ │ │ │ │ ) │
│ 933 │ │ │ config_tokenizer_class = config.tokenizer_class │
│ │
│ /private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/transformers/models/auto/con │
│ │
│ 1130 │ │ │ │ if pattern in str(pretrained_model_name_or_path): │
│ 1131 │ │ │ │ │ return CONFIG_MAPPING[pattern].from_dict(config_dict, **unused_kwargs │
│ 1132 │ │ │
│ ❱ 1133 │ │ raise ValueError( │
│ 1134 │ │ │ f"Unrecognized model in {pretrained_model_name_or_path}. " │
│ 1135 │ │ │ f"Should have a `model_type` key in its {CONFIG_NAME}, or contain one of the │
│ 1136 │ │ │ f"in its name: {', '.join(CONFIG_MAPPING.keys())}" │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Unrecognized model in allenai/OLMoE-1B-7B-0924. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: albert, align, altclip, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, colpali, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deformable_detr, deit, depth_anything, depth_pro, deta, detr, diffllama, dinat, dinov2, dinov2_with_registers, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, emu3, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, git, glm, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hiera, hubert, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mistral3, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_vl, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, seamless_m4t, seamless_m4t_v2, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip_vision_model, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, table-transformer, tapas, textnet, time_series_transformer, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xmod, yolos, yoso, zamba, zamba2, zoedepth
E0401 17:46:26.613000 140547013551936 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 3111472) of binary: /private/home/yashs/.conda/envs/molmo1/bin/python
Traceback (most recent call last):
File "/private/home/yashs/.conda/envs/molmo1/bin/torchrun", line 8, in
sys.exit(main())
File "/private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "/private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
run(args)
File "/private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
elastic_launch(
File "/private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/private/home/yashs/.conda/envs/molmo1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
launch_scripts/eval_downstream.py FAILED
------------------------------------------------------------
Failures:

------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-04-01_17:46:26
host : devfair0645.h2.fair
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3111472)
error_file:
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
```