craigm26 / craigm26/OpenCastor

Phone tools for the chat: every sensor the phone has is a tool the brain can use

Open
#944 0 comments 0 reactions 0 assignees View on GitHub
intelligence marathon-2026-08-15
Dominant language
Python
Stars
28
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Why (operator, verbatim, bench 2026-08-16)
"The phone chat should have access to all the tools that the phone has access to — we proved in Sensors that the phone has depth installed, we just need to use it."

Today the chat only gets what a human manually attaches: one still, one sense-snapshot chip (tilt/heading/location), and the forward view while the mount camera runs. The phone is carrying a depth camera, an IMU, GPS, a barometer, microphones, and six cameras — and the brain can't ask for any of it mid-turn.

## The authority line that makes this easy
Reading the OWNER's OWN device is not robot authority. Phone-sensor reads need no signature and no gateway — they need a visible indicator and the owner-tier trust model (#943). Motion stays signed; sensing is free.

## What
1. A phone-tool registry the chat brain can invoke mid-turn (agentic tool use, answered locally in milliseconds, no upstream round-trip):
- `phone.still` — a frame from a named camera (front/back/ultrawide)
- `phone.clearance` — the ForwardClearance reading (distance ahead, floor state, freshness, tier)
- `phone.depth` — a depth sample from whatever depth hardware exists (rear LiDAR on Pro phones; front TrueDepth on everything with Face ID)
- `phone.motion` — pitch/roll/yaw/g, `phone.location` — coarse fix, `phone.mic_level`
2. Results ride into the turn as provenance-framed sense blocks (same discipline as senses/memories: a reading, with its age, never a fact).
3. Visible indicator whenever a tool fires (the mic/camera dots plus an in-chat chip: "looked ahead: 3.47 m clear").
4. Vision brains compose: "what's in front of you?" -> phone.clearance + phone.still -> the assigned vision model answers with actual data.

## The front-depth nugget
A phone mounted SCREEN-FORWARD points its TrueDepth sensor down the corridor — real depth on a non-Pro phone via AVCaptureDepthDataOutput (already proven in the Sensors tab heatmap), usable as a pure obstacle range gate even without ARKit world tracking. Worth prototyping as a mount option: it would give every Face ID iPhone genuine depth-gated creep.

## Acceptance
- Ask "what do you see ahead?" with the phone mounted: the brain calls phone.clearance + phone.still itself and answers with the measured distance — zero manual attaching.
- Every tool invocation is visibly indicated and lands in the transcript.
- No phone tool can produce motion or reach the gateway: the census (check_no_policy_reimpl.sh) and the authority tests stay untouched.

## Pointers
- ChatView attachedFrame/attachedSense (the manual versions of exactly these tools)
- PhoneSensorInventory (discovery), PhoneEyes/ForwardClearance (clearance), CamerasView depth heatmap (TrueDepth capture already works)
- RobotLocalEngine (where tool-call turns would be answered locally), #943 (owner tier)

Contributor guide

Open the contributing guide

Research direction

Start with ChatView's attachedFrame and attachedSense entry points, then read PhoneSensorInventory, PhoneEyes, ForwardClearance, CamerasView, and RobotLocalEngine. Trace how local tool-call turns could carry provenance-framed sensor results and visible transcript indicators. Done means the listed phone tools can be invoked mid-turn, results are visibly recorded, and the authority census and tests remain untouched.

Written by the indexing model from the issue text.

Assessment

Tech stack
ios, python
Domain
ai, mobile
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.