craigm26 / craigm26/OpenCastor
Phone tools for the chat: every sensor the phone has is a tool the brain can use
- Dominant language
- Python
- Stars
- 28
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Why (operator, verbatim, bench 2026-08-16)
"The phone chat should have access to all the tools that the phone has access to — we proved in Sensors that the phone has depth installed, we just need to use it."
Today the chat only gets what a human manually attaches: one still, one sense-snapshot chip (tilt/heading/location), and the forward view while the mount camera runs. The phone is carrying a depth camera, an IMU, GPS, a barometer, microphones, and six cameras — and the brain can't ask for any of it mid-turn.
## The authority line that makes this easy
Reading the OWNER's OWN device is not robot authority. Phone-sensor reads need no signature and no gateway — they need a visible indicator and the owner-tier trust model (#943). Motion stays signed; sensing is free.
## What
1. A phone-tool registry the chat brain can invoke mid-turn (agentic tool use, answered locally in milliseconds, no upstream round-trip):
- `phone.still` — a frame from a named camera (front/back/ultrawide)
- `phone.clearance` — the ForwardClearance reading (distance ahead, floor state, freshness, tier)
- `phone.depth` — a depth sample from whatever depth hardware exists (rear LiDAR on Pro phones; front TrueDepth on everything with Face ID)
- `phone.motion` — pitch/roll/yaw/g, `phone.location` — coarse fix, `phone.mic_level`
2. Results ride into the turn as provenance-framed sense blocks (same discipline as senses/memories: a reading, with its age, never a fact).
3. Visible indicator whenever a tool fires (the mic/camera dots plus an in-chat chip: "looked ahead: 3.47 m clear").
4. Vision brains compose: "what's in front of you?" -> phone.clearance + phone.still -> the assigned vision model answers with actual data.
## The front-depth nugget
A phone mounted SCREEN-FORWARD points its TrueDepth sensor down the corridor — real depth on a non-Pro phone via AVCaptureDepthDataOutput (already proven in the Sensors tab heatmap), usable as a pure obstacle range gate even without ARKit world tracking. Worth prototyping as a mount option: it would give every Face ID iPhone genuine depth-gated creep.
## Acceptance
- Ask "what do you see ahead?" with the phone mounted: the brain calls phone.clearance + phone.still itself and answers with the measured distance — zero manual attaching.
- Every tool invocation is visibly indicated and lands in the transcript.
- No phone tool can produce motion or reach the gateway: the census (check_no_policy_reimpl.sh) and the authority tests stay untouched.
## Pointers
- ChatView attachedFrame/attachedSense (the manual versions of exactly these tools)
- PhoneSensorInventory (discovery), PhoneEyes/ForwardClearance (clearance), CamerasView depth heatmap (TrueDepth capture already works)
- RobotLocalEngine (where tool-call turns would be answered locally), #943 (owner tier)
Contributor guide
Research direction
Start with ChatView's attachedFrame and attachedSense entry points, then read PhoneSensorInventory, PhoneEyes, ForwardClearance, CamerasView, and RobotLocalEngine. Trace how local tool-call turns could carry provenance-framed sensor results and visible transcript indicators. Done means the listed phone tools can be invoked mid-turn, results are visibly recorded, and the authority census and tests remain untouched.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ios, python
- Domain
- ai, mobile
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100