craigm26 / craigm26/OpenCastor

SmolVLA for Bob: the first learned manipulation skill, through the gap rail

Open
#940 1 comment 0 reactions 0 assignees View on GitHub
intelligence marathon-2026-08-15 stale
Dominant language
Python
Stars
28
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Why
The 2026 open-weights VLA field sorts cleanly for Bob's hardware: SmolVLA (~450M, LeRobot-native, benchmarked on physical SO-101, 60–75% of OpenVLA at 1/14th the size, closing to 5–10% after single-task fine-tuning with ~200 demos; ≥10 Hz on consumer hardware). It is a POLICY under LeRobot, not an Ollama pull — a gap-rail skill, the biggest single project on this list. Expect this issue to spawn sub-issues.

## What (phased — the marathon scopes phase 1, maybe 2)
1. **Record**: teleop demo capture through the app's existing arm controls → LeRobot dataset format on the Pi (every commanded action already flows through the signed gateway; recording is a tap)
2. **Train**: fine-tune SmolVLA off-Pi (rented GPU-hour); document the loop
3. **Serve**: smolvla actuator-side runtime on the Pi emitting joint targets INTO the gateway path — every action judged, tier-gated, receipted, deadman-covered like any other; declared via `castor capability add` (operator-signed)

## Acceptance (phase 1)
- 10 recorded demos of one pick-place task replayable from the dataset; format loads in LeRobot
## Authority rule
The policy drafts motion; the gateway remains the exclusive path to the servo bus. A VLA that could bypass the gateway is a non-starter — same rule as everything else in this stack.

## Pointers
- castor/hardware/so_arm101/* (safety_bridge, rcan_bridge), docs/SKILL-GAPS.md (the rail), Bob gripper manifest-zero lesson (calibration_api docstring)

Contributor guide

Open the contributing guide

Research direction

Read castor/hardware/so_arm101/*, especially safety_bridge and rcan_bridge, along with docs/SKILL-GAPS.md and the calibration_api docstring lesson. Start by tracing the app's existing arm controls and how commanded actions reach the gateway, then determine how phase 1 recording maps to LeRobot format. Done means 10 demos of one pick-place task are replayable from a dataset that loads in LeRobot.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning, python, raspberry-pi
Domain
embedded-iot, machine-learning, robotics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.