craigm26 / craigm26/OpenCastor

Route the per-robot Vision role to local vision models (gemma4:12b)

Open
#939 1 comment 0 reactions 0 assignees View on GitHub
intelligence marathon-2026-08-15
Dominant language
Python
Stars
28
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Why (bench evidence, 2026-08-14)
gemma4:12b-it-qat is the first LOCAL model on the bench that can genuinely see (encoder-free multimodal — verified: read a dense QR on-device, 135 s cold; the console's Ollama path now forwards image_b64). But the iOS per-robot Vision assignment only knows how to route to Gemini Robotics (useVisionBrainForImages). A local vision option exists and cannot be picked.

## What
- BrainAssignments Vision role accepts a robotLocal pick with a model name; ChatEngineProvider routes image turns to that model via /models/chat req.model override (no active-model flip — the pattern chat picks already use)
- Detected-brains list marks vision-capable local models (suggestions ladder already tags gemma4:12b as the VISION brain; consider surfacing Ollama's own vision capability flag via /models/local)
- Latency honesty in UI: local vision ≈ a minute per look on a Pi — say so where the pick is made

## Acceptance
- Assign Vision=gemma4:12b on the rover; attach a photo in chat with qwen on chat duty → the photo answer comes from the 12B, words from qwen; airplane-mode Pi-LAN still works
## Pointers
- models_api.py image wiring (2026-08-14), BrainAssignmentsTests precedence rules, RobotLocalEngine useVisionBrainForImages seam

Contributor guide

Open the contributing guide

Research direction

Start with the image wiring in models_api.py, then read the precedence rules in BrainAssignmentsTests and the RobotLocalEngine useVisionBrainForImages seam. Trace the /models/chat model override and local-model detection through the assignment flow. Done means Vision=gemma4:12b answers attached photos while qwen handles text, including in airplane-mode Pi-LAN use.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python, raspberry-pi
Domain
ai, backend-api-design, mobile-dev, robotics
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.