cactus-compute / cactus-compute/cactus

Gemma 4 vision_encoder.mlpackage not working as intended

Open
#592 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6k
Forks
501
Avg merge
1d 18h
Merged PRs (30d)
4

Description

I ran 2 vision tests with the Gemma 4 model with the same picture. One with the vision_encoder.mlpackage loaded and the other with the CPU vision encoder fallback. The CPU vision encoder was able to process the image properly while the vision_encoder.mlpackage made the model hallucinate.

## CPU Vision Encoder Results
```
'This image appears to be a modern, well-lit meeting or classroom space.\n\nHere is a breakdown of what I see:\n\n**Main Elements:**\n\n* **Setting:** It is a contemporary meeting room, likely designed for presentations or collaborative work.\n* **Focal Point (Whiteboard):** The dominant feature is a large, expansive **whiteboard** that covers an entire wall.'
```

## MLPackage Vision Encoder Results
```
[WARN] [npu] [CactusANE] multi input name=hidden_states shape=(
2520,
768
) strides=(
768,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi input name=cos_full shape=(
2520,
1,
64
) strides=(
64,
64,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi input name=sin_signed shape=(
2520,
1,
64
) strides=(
64,
64,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi input name=attention_mask shape=(
1,
2520,
2520
) strides=(
6350400,
2520,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi output name=output shape=(
2520,
768
) strides=(
768,
1
) dtype=65552

'This image is a **word cloud** or a **collection of text/symbols** that appears to be in multiple languages, though the density and mix make a definitive interpretation difficult without more context.'
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two vision tests described in the issue, comparing the vision_encoder.mlpackage path with the CPU vision encoder fallback. Trace the mlpackage execution and the logged NPU inputs and outputs, then verify that the packaged encoder produces the same image understanding as the CPU path without hallucinated content.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
computer-vision, machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.