cactus-compute / cactus-compute/cactus
Gemma 4 vision_encoder.mlpackage not working as intended
- Dominant language
- C++
- Stars
- 6k
- Forks
- 501
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 4
Description
I ran 2 vision tests with the Gemma 4 model with the same picture. One with the vision_encoder.mlpackage loaded and the other with the CPU vision encoder fallback. The CPU vision encoder was able to process the image properly while the vision_encoder.mlpackage made the model hallucinate.
## CPU Vision Encoder Results
```
'This image appears to be a modern, well-lit meeting or classroom space.\n\nHere is a breakdown of what I see:\n\n**Main Elements:**\n\n* **Setting:** It is a contemporary meeting room, likely designed for presentations or collaborative work.\n* **Focal Point (Whiteboard):** The dominant feature is a large, expansive **whiteboard** that covers an entire wall.'
```
## MLPackage Vision Encoder Results
```
[WARN] [npu] [CactusANE] multi input name=hidden_states shape=(
2520,
768
) strides=(
768,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi input name=cos_full shape=(
2520,
1,
64
) strides=(
64,
64,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi input name=sin_signed shape=(
2520,
1,
64
) strides=(
64,
64,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi input name=attention_mask shape=(
1,
2520,
2520
) strides=(
6350400,
2520,
1
) dtype=65552
[WARN] [npu] [CactusANE] multi output name=output shape=(
2520,
768
) strides=(
768,
1
) dtype=65552
'This image is a **word cloud** or a **collection of text/symbols** that appears to be in multiple languages, though the density and mix make a definitive interpretation difficult without more context.'
```
Contributor guide
Research direction
Start by reproducing the two vision tests described in the issue, comparing the vision_encoder.mlpackage path with the CPU vision encoder fallback. Trace the mlpackage execution and the logged NPU inputs and outputs, then verify that the packaged encoder produces the same image understanding as the CPU path without hallucinated content.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- computer-vision, machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100