lablup / lablup/mlxcel

fix(granite_vision): descriptive prompts draw a refusal, colour questions work

Open
#1,683 1 comment 0 reactions 0 assignees View on GitHub
area:models modelsize:small modeltype:vlm priority:high status:ready type:bug
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Problem

`granite-vision-3.2-2b-4bit` expands the `` placeholder into the prompt and then answers as if no image was supplied.

```
./target/release/mlxcel generate -m models/mlx/granite-vision-3.2-2b-4bit -p "What is in this image? Describe it briefly." --image tests/fixtures/test_image.png -n 48
```

Observed on M1 Ultra (mlxcel 30ab5a39, MLX pin 9a795735, release build with `metal,accelerate`):

```
Granite Vision: expanded 1 placeholder(s) (1485 total image tokens)
What is in this image? Describe it briefly.I'm sorry, but I can't provide a description of the image as I can't see it. Please provide a description of the image or upload it for me to describe.
[Generated 36 tokens in 1.64s = 21.96 tok/s]
```

The same command on M5 Max produces the same result (31 tokens, 71.45 tok/s, same refusal text), so this is not host-specific. `tests/fixtures/test_image.png` is a 224x224 solid orange square, and mlx-vlm 0.6.17 on the same checkpoint with the same image answers "The image is a uniformly colored square with a vibrant orange hue", so the checkpoint is fine and the vision information is recoverable from that image. The token slots exist (1485 image tokens inserted, 1543 measured prompt tokens); the model behaves as though those positions carry no visual content.

A check that reads `prompt_tokens` sees a large, healthy-looking number here, so this class is invisible to any length-based gate and shows up only in the generated text. The distinct `FAIL:image_not_applied` case, where the image never expands at all, is already caught at `scripts/bench_mlxlm.py:391`; this one is not.

## Starting points

Verified by reading the tree, so do not re-derive:

- Expansion and injection are one code path. `src/multimodal/vlm_runtime.rs:2003-2032` is a single match arm that calls `insert_granite_vision_image_tokens` and then `granite.get_input_embeddings(...)` on the expanded ids, so "the injection is never called" is ruled out.
- The counts agree by construction. `insert_granite_vision_image_tokens` (`src/multimodal/granite_vision_prompt.rs:34`) and `pack_image_features` (`src/vision/granite_vision.rs:124`) both derive from `feature_side` and `base_tokens`, and 1485 equals 729 + 27*28 for one 224x224 image at `feature_side` 27, which is the packer's own row count.

Unverified leads, offered as a search order:

- Numerics inside `GraniteVisionVLModel::get_input_embeddings` (`src/vision/granite_vision.rs:67`): the multi-tap SigLIP concat, the projector, the AnyRes re-tiling and unpad in `pack_image_features`, or the scatter in `merge_llava` / `masked_scatter` (`src/vision/merge.rs:175` and `src/vision/merge.rs:273`).
- Magnitude mismatch across the merge boundary: `GraniteModel::forward_embeds` (`src/models/granite.rs:343`) applies `embedding_multiplier` to the merged stream and `get_input_embeddings` must not pre-scale, so a disagreement leaves text fluent while image rows sit at the wrong scale.
- Loader-supplied inputs in `src/loading/vlm_granite_vision.rs` (`vision_feature_layers`, `image_newline`, `image_token_index`, AnyRes pinpoints), where a wrong tap list or mis-keyed `image_newline` degrades features without changing any count.
- The `slice_update` functional-return class in `src/lib/mlxcel-core/src/layers.rs` produced this exact symptom in Granite 4 Vision before, but neither `src/vision/granite_vision.rs` nor `src/vision/merge.rs` calls `slice_update` today, so it looks already ruled out here; confirm cheaply rather than assuming.

## Scope

**In scope:** find the stage where mlxcel diverges from mlx-vlm 0.6.17 on this checkpoint, fix it, add a regression test at that stage.

**Out of scope:** a generation-content check for `scripts/bench_mlxlm.py` that would catch this class; file a separate issue and link it here.

## Acceptance criteria

- [ ] The command above produces a description identifying the image content (a solid orange square) instead of a refusal, on Apple Silicon with `metal,accelerate`.
- [ ] The PR names the diverging stage, located by a stage-by-stage numerical comparison against mlx-vlm 0.6.17 on the same checkpoint and image.
- [ ] A test covering the fixed stage exists and fails against current `main`.
- [ ] Validated through the real CLI generate path, not only a unit test on the helper.

## Verification

```
cargo build --release --features metal,accelerate
./target/release/mlxcel generate -m models/mlx/granite-vision-3.2-2b-4bit -p "What is in this image? Describe it briefly." --image tests/fixtures/test_image.png -n 48
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check
```

A pass is generated text naming the image content; a refusal, or a description that does not follow from the image, is a fail.

Contributor guide

Open the contributing guide

Research direction

Start with the Granite Vision path in src/multimodal/vlm_runtime.rs:2003-2032, src/vision/granite_vision.rs, and src/vision/merge.rs, comparing each stage numerically with mlx-vlm 0.6.17 on the stated checkpoint and image. Add a regression test at the diverging stage, then validate through the real CLI command and the listed workspace checks; done means it describes the solid orange square rather than refusing.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
computer-vision
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.