feat(detection): route vision-stripped VLM checkpoints to a text-only path
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
## Summary
Community checkpoints increasingly ship a VLM `model_type` with the vision tower stripped: the weights hold only `language_model.*` and the config either omits `vision_config`, sets it to `{}` / `null`, or keeps it and sets `language_model_only: true`. mlxcel handles the first shape for four families (`qwen3_5`, `qwen3_5_moe`, `gemma3`, `gemma4`) through `detect_text_or_vlm`, and fails on every other shape: an empty or null `vision_config` still counts as "has vision" and the VLM loader then errors on the sub-config, `language_model_only: true` is a deliberate named error, and the Qwen2-VL / Qwen2.5-VL / Qwen3-VL / Qwen3-VL-MoE families have no text-only route at all. This change makes detection key on the presence of vision weights and makes the Qwen VL family loaders tolerate a missing tower.
## Current behavior
`src/models/detection.rs:55-57`:
```rust
pub(crate) fn has_vision_config(config: &serde_json::Value) -> bool {
config.get("vision_config").is_some()
}
```
so `"vision_config": {}` and `"vision_config": null` route to the VLM loader, which then fails in `parse_required_vlm_subconfig` (`src/loading/vlm.rs:177-187`, `Failed to parse ... vision config`) or in the encoder constructor on the first missing `vision_tower.*` tensor (`src/loading/vlm_qwen.rs:260-272` for `qwen3_vl`, `:212-226` for `qwen2_5_vl`, `:56-75` for `qwen2_vl`, `:306-318` for `qwen3_vl_moe`).
`src/models/qwen3_5.rs:236-251` `validate_qwen35_wrapper_config` returns `Qwen35UnsupportedConfig::LanguageModelOnly` for `language_model_only: true`, even when the checkpoint is exactly the vision-stripped build the flag describes. The text loader for the same family already strips `vision_tower.*` / `model.visual.*` and accepts `language_model.*` keys (`src/models/qwen3_5.rs:3140-3158` `sanitize_moe_weights`, `src/loading/special.rs:44-58` `qwen35_text_config`), so the routing is the only obstacle for that family.
`gemma4` already decides by weights (`src/models/detection.rs:59-79` `gemma4_has_vision_weights` reads `model.safetensors.index.json`), which is the pattern to generalize.
A stripped Qwen3.6 build (`leonsarmiento/ThinkingCap-Qwen3.6-27B-3bit-mlx`: `model_type qwen3_5`, no `vision_config`, `language_model_only: false`, weights under `language_model.*` only) loads today because it happens to omit `vision_config`; the same weights with `vision_config: {}` or `language_model_only: true` do not.
## Expected behavior
Detection decides "text-only" for a VLM `model_type` when any of the following holds:
1. `vision_config` is absent, `null`, or an empty object;
2. top-level `language_model_only` is `true`;
3. the safetensors index (or the single `model.safetensors` header) contains no key starting with `vision_tower.`, `model.visual.`, `model.vision_tower.`, `visual.`, or `vision_model.`.
When text-only is decided for `qwen3_5` / `qwen3_5_moe` / `gemma3` / `gemma4`, route to the existing text `ModelType`. For `qwen2_vl`, `qwen2_5_vl`, `qwen3_vl`, `qwen3_vl_moe` (whose text decoders carry MRoPE and cannot reuse `Qwen2` / `Qwen3`), keep the VLM `ModelType` but load with `vision_encoder: None`; an image or video request on such a model returns the error `model was loaded without a vision tower (text-only checkpoint)` instead of panicking, and text requests run on the text-only fast path that already exists (`can_use_text_only_fast_path`).
A config that says `language_model_only: true` but ships vision weights loads text-only (the flag wins) with a one-line stderr notice naming the flag.
## Implementation plan
1. `src/models/detection.rs`: replace `has_vision_config` with `fn vlm_has_vision(config: &Value, model_path: &Path) -> bool` implementing rules 1-3 (reuse `gemma4_has_vision_weights` as the weight scan, generalized to the prefix list above; also scan a single-file `model.safetensors` header when no index exists). Use it in `detect_text_or_vlm` (pass `model_path`) and for the Qwen VL arms to set a new `ModelType` flag or a `LoadOptions { text_only: bool }` carried to the loader.
2. `src/models/qwen3_5.rs:236-251`: `language_model_only: true` stops being an error on the VLM loader because detection no longer sends such configs there; keep the wrong-type error.
3. `src/loading/vlm_qwen.rs`: in `load_qwen2_vl`, `load_qwen2_5_vl`, `load_qwen3_vl`, `load_qwen3_vl_moe`, when `text_only`, skip `parse_required_vlm_subconfig("vision_config")` and the encoder constructor; build the model with `vision_encoder: None` and a processor built from defaults for `spatial_merge_size` (read from `vision_config` when present, else 2).
4. `src/vision/qwen2_vl.rs:36`, `src/vision/qwen2_5_vl.rs:36`, `src/vision/qwen3_vl.rs:38`, `src/vision/qwen3_vl_moe.rs:32`: change `vision_encoder` to `Option<...>`; `get_input_embeddings` / `input_embeddings_multimodal` return `Err` (or the runtime checks `has_vision_tower()` before calling) with the message above. Add `fn has_vision_tower(&self) -> bool` to `VlmRuntimeRef` so `src/multimodal/vlm_runtime.rs` and `src/server/model_worker.rs` reject media requests cleanly; `mlxcel list` / model capabilities report `vision: false` for such a load.
5. Tests in `src/models/detection_tests.rs`: `empty_vision_config_routes_to_text` (`qwen3_5` with `vision_config: {}`), `null_vision_config_routes_to_text`, `language_model_only_routes_to_text`, `no_vision_weights_in_index_routes_to_text` (temp dir with an index holding only `language_model.*` keys), `qwen3_vl_without_vision_config_loads_text_only` (loader-level, synthetic tiny weights).
6. `docs/supported-models.md`: a short paragraph under the Qwen VL and Qwen3.5 entries describing text-only loads, and remove the sentence that `language_model_only: true` is rejected.
## Validation
(a) Unit tests as in step 5.
(b) Real checkpoint: `leonsarmiento/ThinkingCap-Qwen3.6-27B-3bit-mlx` (12.5 GB, stripped Qwen3.6-27B; `model_type qwen3_5`). Edit a local copy's `config.json` to add `"vision_config": {}` and, separately, `"language_model_only": true`, and confirm all three variants load and produce identical text:
```
./target/release/mlxcel generate -m models/ThinkingCap-Qwen3.6-27B-3bit-mlx -p "Explain why the sky is blue in two sentences." -n 64
```
Acceptance: the three config variants produce byte-identical output; an `--image` request on the loaded model returns the text-only error rather than a panic; `mlxcel list` still reports the family.
## Acceptance criteria
- [ ] `vision_config` absent / null / `{}` and `language_model_only: true` all route VLM `model_type`s to a text-only load
- [ ] A missing vision tower in the weight index routes to text-only for the families above
- [ ] Qwen2-VL / Qwen2.5-VL / Qwen3-VL / Qwen3-VL-MoE load with `vision_encoder: None` and reject media requests with a named error
- [ ] Stripped Qwen3.6 checkpoint loads under all three config variants with identical output
- [ ] docs/supported-models.md updated
- [ ] detection table in src/models/detection.rs updated with a test
- [ ] cargo test --workspace --profile test-fast --features metal,accelerate passes
- [ ] cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --all -- --check pass
## Out of scope
- Text-only loads for encoder-based VLM families without a reusable text decoder route (LLaVA, Idefics, Molmo, Pixtral); each needs its own follow-up once a stripped checkpoint exists.
Contributor guide
Research direction
Start with src/models/detection.rs and its detection tests, especially the existing gemma4 weight scan, then trace the Qwen VLM loaders in src/loading/vlm_qwen.rs. Review the four Qwen vision runtime files and src/multimodal/vlm_runtime.rs for the text-only and media-error paths. Done means the listed synthetic tests, real checkpoint validation, documentation update, and workspace test, clippy, and format checks pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100