intel / intel/llm-scaler

[vllm] Muse-Glimmer-30B cannot be served on 0.21.0-b3: AutoProcessor returns TokenizersBackend — Onyx image processor missing from the image

Open
#607 22 comments 0 reactions 1 assignee Claimed by @hzjane View on GitHub
Dominant language
C++
Stars
529
Forks
80
Avg merge
9h 7m
Merged PRs (30d)
38

Description

## Summary

`0.21.0-b3` is announced as *"support Muse-Glimmer-30B"* and `vllm/README.md` lists `meta-models/Muse-Glimmer-30B` with Dynamic Online FP8 ✅. The model cannot be served: the server aborts during startup because no HF processor for `muse_glimmer` / `onyx` is reachable from inside the image.

## Environment

| | |
|---|---|
| Image | `intel/llm-scaler-vllm:0.21.0-b3` |
| vLLM | `0.21.1.dev0+gad7125a43.d20260810` |
| transformers | `5.8.0` |
| torch | `2.11.0+xpu` |
| Host | Ubuntu 26.04 LTS, kernel `7.0.0-29-generic`, `xe` driver |
| GPU | 8× Intel Arc Pro B60 24 GB (2 used, TP=2) |
| Model | `meta-models/Muse-Glimmer-30B` @ `main` (bf16, 59.58 GB, checked out complete) |

## Reproduction

```bash
python3 -m vllm.entrypoints.openai.api_server \
--model /llm/models/Muse-Glimmer-30B \
--tensor-parallel-size 2 \
--quantization fp8 \
--dtype float16 \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 6}' \
--trust-remote-code
```

## Traceback (tail)

```
onyx_mm.py:362 in get_hf_processor -> self.ctx.get_hf_processor(**kwargs)
multimodal/processing/context.py:204 -> cached_processor_from_config(...)
transformers_utils/processor.py:225 in get_processor
TypeError: Invalid type of HuggingFace processor.
Expected type: ,
but found type:
```

## Root cause, verified inside the image

1. transformers 5.8.0 ships no `onyx` / `muse_glimmer` model — `ls transformers/models/ | grep -i 'onyx\|muse'` is empty, so `OnyxProcessor` and its image processor do not exist.
2. vLLM vendors the **config only** — `transformers_utils/config.py:121` `muse_glimmer="OnyxConfig"`; `transformers_utils/processors/` contains just `kimi_k25.py`.
3. The HF repo ships no `.py` at all (weights + json only), so `--trust-remote-code` has nothing to load. The `unsloth/Muse-Glimmer-30B` mirror has none either.
4. Yet `onyx_mm.py:355` requires `proc.image_processor.compute_image_size(...)`.

The vision stack itself is present and complete (`OnyxVisionAttention`, `OnyxVisionBlock`, `OnyxVisionEncoder`, `OnyxVisionAdapter`, plus a `WeightsMapper` covering `model.vision_tower.` / `vision_adapter` / `vision_projection`), so this looks like a packaging gap rather than missing functionality.

## Note: forcing the text-only architecture is not a workaround

`--hf-overrides '{"architectures":["MuseGlimmerForCausalLM"]}'` starts cleanly, passes `/v1/models`, and answers requests — but emits token soup.

The checkpoint uses multimodal naming (`model.language_model.*` — 626 text tensors, `model.vision_tower.*` — 806), and the prefix mapper exists **only** in `onyx_mm.py`:

```python
orig_to_new_prefix={
"model.language_model.": "model.",
...
}
```

`onyx.py` (the text-only `OnyxForCausalLM` path) has no mapper, so zero weights match and the model runs on uninitialized tensors. Nothing is logged about it — `Model loading took 14.26 GiB memory` reports allocated memory, not matched weights. The only hint is `Loading weights took 1.73 seconds` for a 55.46 GiB checkpoint.

Raising a loud error when a checkpoint load matches no weights would have made this immediately obvious.

## Ask

- Ship the Onyx processor (image processor incl. `compute_image_size`) in the image, or document where it is expected to come from.
- Consider failing loudly when a checkpoint load matches zero weights.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.