fix(granite_vision): descriptive prompts draw a refusal, colour questions work
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
## Problem
`granite-vision-3.2-2b-4bit` expands the `` placeholder into the prompt and then answers as if no image was supplied.
```
./target/release/mlxcel generate -m models/mlx/granite-vision-3.2-2b-4bit -p "What is in this image? Describe it briefly." --image tests/fixtures/test_image.png -n 48
```
Observed on M1 Ultra (mlxcel 30ab5a39, MLX pin 9a795735, release build with `metal,accelerate`):
```
Granite Vision: expanded 1 placeholder(s) (1485 total image tokens)
What is in this image? Describe it briefly.I'm sorry, but I can't provide a description of the image as I can't see it. Please provide a description of the image or upload it for me to describe.
[Generated 36 tokens in 1.64s = 21.96 tok/s]
```
The same command on M5 Max produces the same result (31 tokens, 71.45 tok/s, same refusal text), so this is not host-specific. `tests/fixtures/test_image.png` is a 224x224 solid orange square, and mlx-vlm 0.6.17 on the same checkpoint with the same image answers "The image is a uniformly colored square with a vibrant orange hue", so the checkpoint is fine and the vision information is recoverable from that image. The token slots exist (1485 image tokens inserted, 1543 measured prompt tokens); the model behaves as though those positions carry no visual content.
A check that reads `prompt_tokens` sees a large, healthy-looking number here, so this class is invisible to any length-based gate and shows up only in the generated text. The distinct `FAIL:image_not_applied` case, where the image never expands at all, is already caught at `scripts/bench_mlxlm.py:391`; this one is not.
## Starting points
Verified by reading the tree, so do not re-derive:
- Expansion and injection are one code path. `src/multimodal/vlm_runtime.rs:2003-2032` is a single match arm that calls `insert_granite_vision_image_tokens` and then `granite.get_input_embeddings(...)` on the expanded ids, so "the injection is never called" is ruled out.
- The counts agree by construction. `insert_granite_vision_image_tokens` (`src/multimodal/granite_vision_prompt.rs:34`) and `pack_image_features` (`src/vision/granite_vision.rs:124`) both derive from `feature_side` and `base_tokens`, and 1485 equals 729 + 27*28 for one 224x224 image at `feature_side` 27, which is the packer's own row count.
Unverified leads, offered as a search order:
- Numerics inside `GraniteVisionVLModel::get_input_embeddings` (`src/vision/granite_vision.rs:67`): the multi-tap SigLIP concat, the projector, the AnyRes re-tiling and unpad in `pack_image_features`, or the scatter in `merge_llava` / `masked_scatter` (`src/vision/merge.rs:175` and `src/vision/merge.rs:273`).
- Magnitude mismatch across the merge boundary: `GraniteModel::forward_embeds` (`src/models/granite.rs:343`) applies `embedding_multiplier` to the merged stream and `get_input_embeddings` must not pre-scale, so a disagreement leaves text fluent while image rows sit at the wrong scale.
- Loader-supplied inputs in `src/loading/vlm_granite_vision.rs` (`vision_feature_layers`, `image_newline`, `image_token_index`, AnyRes pinpoints), where a wrong tap list or mis-keyed `image_newline` degrades features without changing any count.
- The `slice_update` functional-return class in `src/lib/mlxcel-core/src/layers.rs` produced this exact symptom in Granite 4 Vision before, but neither `src/vision/granite_vision.rs` nor `src/vision/merge.rs` calls `slice_update` today, so it looks already ruled out here; confirm cheaply rather than assuming.
## Scope
**In scope:** find the stage where mlxcel diverges from mlx-vlm 0.6.17 on this checkpoint, fix it, add a regression test at that stage.
**Out of scope:** a generation-content check for `scripts/bench_mlxlm.py` that would catch this class; file a separate issue and link it here.
## Acceptance criteria
- [ ] The command above produces a description identifying the image content (a solid orange square) instead of a refusal, on Apple Silicon with `metal,accelerate`.
- [ ] The PR names the diverging stage, located by a stage-by-stage numerical comparison against mlx-vlm 0.6.17 on the same checkpoint and image.
- [ ] A test covering the fixed stage exists and fails against current `main`.
- [ ] Validated through the real CLI generate path, not only a unit test on the helper.
## Verification
```
cargo build --release --features metal,accelerate
./target/release/mlxcel generate -m models/mlx/granite-vision-3.2-2b-4bit -p "What is in this image? Describe it briefly." --image tests/fixtures/test_image.png -n 48
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check
```
A pass is generated text naming the image content; a refusal, or a description that does not follow from the image, is a fail.
Contributor guide
Research direction
Start with the Granite Vision path in src/multimodal/vlm_runtime.rs:2003-2032, src/vision/granite_vision.rs, and src/vision/merge.rs, comparing each stage numerically with mlx-vlm 0.6.17 on the stated checkpoint and image. Add a regression test at the diverging stage, then validate through the real CLI command and the listed workspace checks; done means it describes the solid orange square rather than refusing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100