lmstudio-ai / lmstudio-ai/mlx-engine

Text-only architectures implemented only in mlx_vlm (e.g. deepseek_v4) fail to load because dispatch checks only for vision_config

Open
#357 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
133
Avg merge
21h 6m
Merged PRs (30d)
1

Description

## Summary

For a text-only model, `mlx_engine/generate.py`'s model-kit selection dispatches purely on `"vision_config" in config_json` (around `load_model`, the `is_batchable()` branch). If that key is absent, it unconditionally calls `mlx_lm_load(model_path, lazy=True)` — even when the architecture in question has no `mlx_lm` implementation at all, but *does* have one in `mlx_vlm`, bundled in the exact same environment.

Concretely, for `deepseek_v4` (a genuinely text-only architecture — no vision component): the bundled environment's `mlx_lm/models/` stops at `deepseek_v32`, so loading fails outright. The *same* bundled environment's `mlx_vlm/models/deepseek_v4/` is fully implemented (including an MTP speculative-decoding drafter), but it's never reached, because the dispatch never considers `mlx_vlm` for a model that lacks `vision_config`.

## Repro

Load any local MLX checkpoint with `model_type: deepseek_v4` (text-only, no `vision_config` key) via `lms load` or the GUI.

Result:
```
ValueError: Model type deepseek_v4 not supported.

At:
.../mlx_lm/utils.py(192): _get_classes
.../mlx_lm/utils.py(348): load_model
.../mlx_lm/utils.py(509): load
.../mlx_engine/generate.py(289): is_batchable
```

## Workaround found

Adding an empty `"vision_config": {}` to the checkpoint's `config.json` routes the same load through the `BatchedVisionModelKit` → `mlx_vlm` path instead, which has the architecture and loads/generates correctly (confirmed end-to-end, including at extended context lengths). `mlx_vlm.utils.load_image_processor` degrades gracefully for a model class with no `ImageProcessor` attribute, so this doesn't require faking any real vision capability — it's purely a dispatch trick, and a fragile one for users who don't know to do it.

## Suggested fix

Before deciding "no `vision_config` → use `mlx_lm`", check whether `mlx_lm` actually has an implementation for the model's `model_type` (a cheap `_get_classes`-style lookup) and fall back to `mlx_vlm` if not, rather than dispatching solely on the presence of a config key that's orthogonal to which package actually implements the architecture. This matters more as new architectures increasingly land in `mlx_vlm` first even when text-only.

## Environment

- LM Studio 0.4.20 (build 1), macOS
- mlx-llm backend (nax-advsimd), engine build `app-mlx-generate-mac26-arm64@33`
- Model: local MLX conversion of `deepseek-ai/DeepSeek-V4-Flash` (`deepseek_v4` architecture, text-only)
- Related: #356 (the same underlying gap also causes the manual context-length control to be capped, since the auto-fit path that would otherwise report a real value is never reached for this architecture either)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mlx_engine/generate.py around load_model and the is_batchable() dispatch, then inspect the model-type lookup behavior described for mlx_lm and mlx_vlm. Reproduce with a local deepseek_v4 checkpoint lacking vision_config through lms load or the GUI. Done means text-only architectures implemented only in mlx_vlm load without a vision_config workaround, while existing vision-model routing still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.