craigm26 / craigm26/OpenCastor
Route the per-robot Vision role to local vision models (gemma4:12b)
- Dominant language
- Python
- Stars
- 28
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Why (bench evidence, 2026-08-14)
gemma4:12b-it-qat is the first LOCAL model on the bench that can genuinely see (encoder-free multimodal — verified: read a dense QR on-device, 135 s cold; the console's Ollama path now forwards image_b64). But the iOS per-robot Vision assignment only knows how to route to Gemini Robotics (useVisionBrainForImages). A local vision option exists and cannot be picked.
## What
- BrainAssignments Vision role accepts a robotLocal pick with a model name; ChatEngineProvider routes image turns to that model via /models/chat req.model override (no active-model flip — the pattern chat picks already use)
- Detected-brains list marks vision-capable local models (suggestions ladder already tags gemma4:12b as the VISION brain; consider surfacing Ollama's own vision capability flag via /models/local)
- Latency honesty in UI: local vision ≈ a minute per look on a Pi — say so where the pick is made
## Acceptance
- Assign Vision=gemma4:12b on the rover; attach a photo in chat with qwen on chat duty → the photo answer comes from the 12B, words from qwen; airplane-mode Pi-LAN still works
## Pointers
- models_api.py image wiring (2026-08-14), BrainAssignmentsTests precedence rules, RobotLocalEngine useVisionBrainForImages seam
Contributor guide
Research direction
Start with the image wiring in models_api.py, then read the precedence rules in BrainAssignmentsTests and the RobotLocalEngine useVisionBrainForImages seam. Trace the /models/chat model override and local-model detection through the assignment flow. Done means Vision=gemma4:12b answers attached photos while qwen handles text, including in airplane-mode Pi-LAN use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, python, raspberry-pi
- Domain
- ai, backend-api-design, mobile-dev, robotics
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100