lablup / lablup/mlxcel

feat(xla): support Qwen3-VL and DeepStack vision features

Open
#868 0 comments 0 reactions 0 assignees View on GitHub
area:inference area:models priority:low status:review type:enhancement
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Parent and dependencies

- Parent epic: #566
- Blocked by #865 and #867
- Existing MLX implementation: `src/vision/qwen3_vl.rs`, `src/vision/qwen3_vl_moe.rs`, `src/vision/encoders/qwen3_vl.rs`, `src/models/qwen3_vl.rs`, and `src/models/qwen3_vl_moe.rs`

## Problem

Qwen3-VL shares M-RoPE concepts with Qwen2-VL but has a materially different vision tower. It removes Qwen2.5's windowed attention, uses different cumulative-sequence semantics and fast position-embedding interpolation, and returns DeepStack side features from configured vision blocks. Those features are projected with separate merger modules and injected into selected language layers.

An implementation that only ports the final vision merger will load and generate but omit part of the checkpoint architecture.

## Goal

Support pinned dense and, where configuration-compatible, MoE Qwen3-VL checkpoints end to end on XLA, including the full vision output set and per-layer DeepStack injections.

## Required implementation

- Emit Qwen3-VL patch embedding, learned/fast-interpolated position embeddings, 2D vision RoPE, packed full-attention blocks, MLP/norm variants, main patch merger, and every configured DeepStack merger through StableHLO/IREE.
- Follow Qwen3's `cu_seqlens = h * w` per frame semantics exactly; do not reuse Qwen2.5's spatial-merge-unit/window cumulative lengths.
- Make the vision graph return the main projected tokens plus ordered DeepStack projected features. Bind `deepstack_visual_indexes`, merger count/shapes, patch/temporal/merge sizes, position-table metadata, vision depth, and static grid limits into artifact compatibility identity.
- Construct expanded token positions, M-RoPE coordinates, visual-position indices, target language-layer indices, and per-layer compact feature payloads through the canonical contracts from #864 and #867.
- Match the current MLX ordering and injection point exactly. Validate that each DeepStack output has the same visual-token count/order as the main merger and the placeholder positions.
- Support the dense Qwen3 text decoder first. Enable Qwen3-VL-MoE only when the existing XLA Qwen3 MoE text emitter passes the same prefill/decode oracle; do not claim it from shared config names alone.
- Integrate filtered loading, capability detection, CLI, server, batching, cancellation, and no-media text fallback without a duplicate full MLX decoder or silent MLX vision fallback.
- Treat image and video support separately in validation. Advertise video only after temporal grids and multi-frame position state pass an oracle fixture.

## Non-goals

- Reintroducing Qwen2.5 windowed attention into Qwen3.
- Generalizing DeepStack into arbitrary cross-attention or decode-time side inputs.
- Supporting unbounded visual shapes without compiled buckets.

## Validation

- Pin one dense Qwen3-VL checkpoint/revision and, if enabled, one Qwen3-VL-MoE revision.
- Compare processed patches, fast position embeddings, vision RoPE, cumulative boundaries, selected vision-block states, main merger output, every DeepStack merger output, injection positions/layers, language hidden states after injection, logits/KV, and greedy tokens against MLX and an independent oracle.
- Include no-media text, one image, multiple media, a temporal/video grid when advertised, bucket-boundary/overflow grids, invalid deepstack config, cancellation, and mixed-slot delta/injection isolation.
- Add a negative fixture that drops DeepStack features and proves the intermediate/final oracle detects the architectural omission.
- Run CLI and streaming continuous-batch server paths on a production-relevant IREE target, plus all #865/#867/text-only gates.

## Acceptance criteria

- [ ] Qwen3-specific non-windowed sequence and position semantics are implemented rather than aliased to Qwen2.5.
- [ ] The IREE vision graph returns the main output and every configured DeepStack branch in stable order.
- [ ] Per-layer injections match MLX intermediate hidden states within documented tolerances.
- [ ] Dense deterministic output is token-exact; MoE is advertised only after its own oracle passes.
- [ ] Media capability accurately reflects validated image/video coverage.
- [ ] Cancellation and slot reuse clear both M-RoPE and DeepStack state.
- [ ] No-media Qwen3 text and prior Qwen family regressions remain green.

Contributor guide

Open the contributing guide

Research direction

Start with the existing MLX implementations in src/vision/qwen3_vl.rs, src/vision/qwen3_vl_moe.rs, src/vision/encoders/qwen3_vl.rs, src/models/qwen3_vl.rs, and src/models/qwen3_vl_moe.rs, then read the canonical contracts from #864 and #867. Run the #865, #867, and text-only gates before implementing the XLA/IREE path. Done means validated dense image support with ordered DeepStack outputs and injections, while video and MoE are advertised only after their specified oracle checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.