fix(youtu_vl): multi-window images still described wrong after the patch-order fix
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
## Problem / Background
#1610 fixed the Youtu-VL image processor's patch emission order (raster with `(c, dy, dx)` features, now merge-block-major with `(dy, dx, c)`). That fix is confirmed correct by the single-window control fixture, which went from being described as black to being described as orange. It is not sufficient: both multi-object fixtures still get a description that has nothing to do with their contents, so at least one further defect remains in this family's vision path.
This issue records what was observed so the next person does not have to re-derive it. It does not claim a diagnosis.
## Evidence
Measured on `models/mlx/youtu-vl-4b-instruct` (`tencent/Youtu-VL-4B-Instruct`), M1 Ultra, release build with `metal,accelerate`, greedy (`-t 0 -n 48`), at `fix/issue-1610-youtu-vl-patch-order`.
`models/mlx/youtu-vl-4b-instruct` has `patch_size=16`, `spatial_merge_size=2` and `window_size=256`, so `vit_merger_window_size` is 8 merged units and `smart_resize` snaps every edge to a multiple of 32. The merged grid is therefore one eighth of the resized pixel edge.
| fixture | resized edge | merged grid | windows | mlxcel says, after #1610 | correct? |
|---|---|---|---|---|---|
| `tests/fixtures/test_image.png` (224x224 solid orange) | 224 | 7x7 | one 8x8 window, `get_window_index` is the identity | "The image is a solid, uniform block of bright orange color. There are no visible objects, textures, patterns, or variations — it is a pure, flat field of orange with no discernible features or details." | yes |
| `tests/fixtures/test_image_shapes_336.png` (red square, blue circle, green triangle on light grey) | 352 | 11x11 | 2x2 windows, last row and column partial | "The image contains a single white circle on a black background." | no |
| `tests/fixtures/test_image_shapes.png` (the same scene at 448x448) | 448 | 14x14 | 2x2 windows, last row and column partial | "The image contains a single, solid black circle on a plain white background." | no |
For comparison, the same three runs before #1610 gave "completely black ... a solid black square or rectangle" for the orange fixture and "a single, solid black circle on a white background" for both shape fixtures.
## Observations worth keeping
The one fixture that is correct is the only one whose merged grid fits inside a single attention window. Both fixtures that need more than one window are wrong, and they are wrong in different ways from each other (the 336 rendering inverts the polarity relative to the 448 rendering), which is not what a size-independent defect in the embedding or the merger would produce.
Both wrong answers collapse three objects in three colors into one achromatic object. That is the signature of tokens carrying content from the wrong spatial location rather than of content being erased: the model is confidently describing something, it is just not this image.
Candidate areas, in the order the evidence points at them, none of them verified:
- `window::get_window_index` and the `cu_window_seqlens` it returns in `src/vision/encoders/youtu_vl.rs`, including how partial windows at the right and bottom edges are padded and how the segment boundaries are deduplicated. 11x11 and 14x14 both leave a partial window; 7x7 does not.
- The interaction between the window reordering and `fullatt_block_indexes` in `YoutuVLVisionEncoder::forward_with_spatial`: window-ordered hidden states are fed to the full-attention layers with `full_cu` boundaries, so the two orderings have to agree about what a segment is.
- The window inverse itself is out of scope here. #1600 / #1603 fixed it to `argsort(window_index)` and that fix is at HEAD; it cannot explain the 224 result either way, since the permutation is the identity at that size.
## Reproduce
```
cargo build --release --features metal,accelerate
./target/release/mlxcel generate -m models/mlx/youtu-vl-4b-instruct --image tests/fixtures/test_image_shapes.png -p "What shapes and colors are in this image? Answer briefly." -n 48 -t 0
./target/release/mlxcel generate -m models/mlx/youtu-vl-4b-instruct --image tests/fixtures/test_image_shapes_336.png -p "What shapes and colors are in this image? Answer briefly." -n 48 -t 0
```
`tests/youtu_vl_parity.rs`, added in #1610, already carries both of these as `#[ignore]`d content assertions written at the full correct answer and labelled as known failing against this issue. Fixing this issue is what turns them green; do not weaken them.
## Acceptance Criteria
- [ ] Root cause identified and named, with the reasoning that rules out the alternatives above.
- [ ] `test_image_shapes.png` and `test_image_shapes_336.png` are both described with all three shapes and all three colors by the real checkpoint, with before and after text recorded in the PR.
- [ ] The 224x224 control still reads as orange, so the fix does not trade one size for another.
- [ ] The two known-failing tests in `tests/youtu_vl_parity.rs` are un-gated: their `#[ignore]` reason drops the "known failing" clause and keeps only the checkpoint requirement.
- [ ] A checkpoint-free unit test covers whatever the defect turns out to be, on a grid that spans more than one window.
## Technical Considerations
Related: #1610 (the patch emission order, which this issue is the residual of), #1600 / #1603 (the window inverse in the same encoder), #1611 (the processor's patch-count cap), #1596 / #1601 (the Qwen2.5-VL pair of defects this family mirrors).
Contributor guide
Research direction
Build with `cargo build --release --features metal,accelerate` and reproduce both failures using the commands in the issue. Read `src/vision/encoders/youtu_vl.rs`, focusing on `window::get_window_index`, `cu_window_seqlens`, and `YoutuVLVisionEncoder::forward_with_spatial`, then inspect the ignored assertions in `tests/youtu_vl_parity.rs`. Done means both multi-window fixtures describe all shapes and colors, the orange control remains correct, the assertions are ungated, and a checkpoint-free multi-window unit test covers the defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 46/100