Hosted Teleop Overview

Cloud-relayed VR/keyboard teleop. Operator's browser → broker (CF SFU) → robot, all over WebRTC. No port forwarding, no LAN, no public IP — the robot is just an outbound HTTPS/UDP client.

Two ways to wire it today:

Transport blueprints (preferred) — the broker DataChannels/video are DimOS transports; the robot driver and the hosted control plane live in one module on one process → one CF session. teleop-hosted-go2-transport.

Legacy module wrapper — HostedTeleopModule owns the whole WebRTC + control plane itself. Still shipped (teleop-hosted-go2, teleop-hosted-xarm7) but the transport swap is preferred for Go2.

1. Big picture

┌─────────────┐    HTTPS    ┌───────────────────┐   HTTPS+REST   ┌──────────────┐
│ Operator    │ ──────────► │ dimensional-teleop│ ─────────────► │ Cloudflare   │
│ (browser)   │   X-Bearer  │   broker (EC2)    │   CF app key   │ Realtime SFU │
└──────┬──────┘             │   FastAPI         │                └──────┬───────┘
       │                    └───────────────────┘                       │
       │  HTTPS (X-Robot-API-Key) ────┘                                  │
       │                                                                 │
       │           WebRTC (data + video) ◄──────────────────────────────►│
       ▼                                                                 ▼
   Both peers connect to CF; CF bridges them.            ┌──────────────────────────┐
   Broker only handles SETUP — once channels are         │ Go2HostedConnection      │
   bridged, broker is off the data path.                 │ (driver + control plane) │
                                                         └──────────────────────────┘

Two roles for the broker:

Auth (operator JWT login + robot API key validation).
Session setup (creates CF sessions, bridges DataChannels, pulls video).

Everything operational (PoseStamped, Joy, Twist, video, telemetry, robot commands) flows direct: operator's browser ↔ Cloudflare edge ↔ robot. Broker is not a WebRTC peer.

2. One CF session (why the consolidation matters)

The CF broker provider is a per-process singleton keyed by config, and the Go2 driver runs dedicated_worker=True (its own process). So all hosted broker transports (cmd, video, state, state_back) must live on that same module to share one CF session.

A separate control-plane module would land in a different worker → a second CF session the operator never sees (this is exactly why battery/telemetry used to silently never arrive). The fix: fold the hosted state plane into the driver as Go2HostedConnection(GO2Connection) — one module, one process, one session.

3. The module — transport path (`Go2HostedConnection`)

A dedicated_worker subclass of GO2Connection that adds the hosted state plane and a camera mux. Ports:

class Go2HostedConnection(GO2Connection):
    state_json:    In[bytes]        # operator → robot control JSON (state_reliable)
    cmd_raw:       In[bytes]        # operator → robot command bytes (stats tap)
    cam2_in:       In[Image]        # extra camera (RealSense) for the mux
    video_stats:   Out[VideoStats]  # operator video health, for recorders
    telemetry_out: Out[bytes]       # robot → operator telemetry + acks (state_reliable_back)
    mux_image:     Out[Image]       # composited cam1(Go2)+cam2 → video transport

Threads / sync primitives in this module:

Thread	Runs	Cadence
`Go2HostedTelemetry`	snapshot cmd-plane stats + battery SOC → JSON push	`telemetry_hz` (3 Hz)
per-command worker	`sport_command` / `set_rage_mode` (blocking `publish_request`)	one-shot

_cmd_stats: LiveStreamStats — rolling window over the inbound twist wire.
_stop_event: threading.Event — telemetry loop watches it; stop() joins.
_cam_lock: threading.Lock — guards _cam_frames / _cam_selected; the cv2 resize + hstack composite runs outside the lock.

Robot commands run on a worker thread on purpose: publish_request blocks the shared WebRTC/video loop, so running a gesture like Hello inline froze both cameras. Off-thread → video keeps flowing while the command executes.

Legacy path (`HostedTeleopModule`) — still shipped

The original abstract module owning its own aiortc PC + 4 threads (_loop_thread asyncio, _heartbeat_thread 1 Hz, _telemetry_thread 3 Hz, _control_loop_thread 50 Hz). Concrete subclasses HostedArmTeleopModule / HostedTwistTeleopModule. Used by teleop-hosted-xarm7 and teleop-hosted-go2.

4. Transports — what flows where

Channel	Direction	Reliability	Carries
`cmd_unreliable`	operator → robot	unordered, no retransmits	LCM `TwistStamped` (drive); also tapped for stats + recorder
`state_reliable`	operator → robot	ordered, reliable	JSON: `ping`, `clock_report`, `video_stats`, `sport_cmd`, `set_mode`, `camera_select`
`state_reliable_back`	robot → operator	ordered, reliable	JSON: `pong`, `robot_telemetry` (cmd-plane + SOC), `cmd_ack`
video track	robot → operator	unreliable, paced by camera	H.264 frames (single cam, or mux'd cam1+cam2)

All three datachannels share one SCTP association (MAX_BUNDLE). SCTP ids are assigned by CF.

Inbound demux (transport path): each DataChannel is its own DimOS transport, so payloads arrive already routed by channel. Within a channel we try-decode (JSON for the state plane; LCM TwistStamped for the command tap) rather than keying a global fingerprint dict — foreign/undecodable frames are skipped.

Transport wiring (Go2 transport blueprint)

.transports({
    ("cmd_vel", Twist):          CloudflareTransport.spec("cmd_unreliable", TwistStamped),
    ("color_image", Image):      CloudflareVideoTransport.spec(),
    ("state_json", bytes):       CloudflareTransport.spec("state_reliable"),
    ("telemetry_out", bytes):    CloudflareTransport.spec("state_reliable_back"),
    ("cmd_raw", bytes):          CloudflareTransport.spec("cmd_unreliable"),   # stats tap
    ("cmd_vel_stamped", TwistStamped): CloudflareTransport.spec("cmd_unreliable", TwistStamped),  # recorder tap
})

Multicam swaps the video bind to mux_image and wires cam2_in from the RealSense over LCM.

5. Robot commands (operator → robot)

New on the state_reliable channel. Each is allow-listed robot-side and acked back on state_reliable_back ({type: cmd_ack, nonce, ok}).

Operator UI	JSON	Robot-side
Posture — StandReady, Sit, StandDown, RecoveryStand	`{type: sport_cmd, name, nonce}`	`StandReady` = standup→settle→balance combo; others via allow-list → `sport_command(api_id)`
Actions — Shake Hand, Stretch, FrontPounce, FrontJump	`{type: sport_cmd, name, nonce}`	`ALLOWED_SPORT_CMDS` → `sport_command(api_id)` (worker thread)
E-stop — Damp	`{type: sport_cmd, name: "Damp", nonce}`	firmware damp/limp; reliable channel, out-of-band of twist
Speed bar — Normal / High / Rage	`{type: set_mode, mode, nonce}`	normal/high = browser scale only; rage toggles firmware FSM (`set_rage_mode`, bidirectional)
Camera tabs — cam1 / cam2 / side-by-side	`{type: camera_select, cams}`	`_set_cam_selection` → re-composite the mux

Allow-list is robot-side: an unknown/disallowed name is rejected with ok=false. Acrobatic actions (Pounce/Jump) currently execute on a single click — confirm-gating before unsupervised driving is on the roadmap.

6. Telemetry & stats (robot → operator)

The robot computes command-plane health and forwards it (compute-and-forward — it doesn't consume the stats locally):

robot_telemetry = { type, cmd: {latency_ms, jitter_ms, rate_hz, throughput_bps},
                    soc, robot_ts }

cmd from LiveStreamStats.snapshot() — rolling window over inbound twists. One-way latency uses the twist header send-stamp (clock-sync calibrated).
soc — battery %, cached by GO2Connection._on_lowstate from bms_state.soc.
Command loss is not currently measured (would need an operator send-count; seq was dropped as ROS-incompatible / fragile to decode). The loss_pct / reorder_count helpers remain in stream_stats.py for when it's wired.

Operator HUD (web/js/hud.js) renders this as a DOM pill + in-VR stats quad with a green/amber/red dot over video + cmd-plane + RTT; the cockpit (web/js/views/go2.js) shows battery + telemetry summary/detail.

7. Multi-camera

Go2HostedConnection composites the Go2 camera (cam1) and a RealSense (cam2_in → cam2) into one video track. Operator tabs select cam1 / cam2 / side-by-side via camera_select; _composite cv2-resizes to a common height and np.hstackes the tiles. Graceful degrade: with only cam1 present it ships the single frame. (Known: RGB/BGR mismatch in the side-by-side tile — RealSense RGB vs Go2 BGR — not yet corrected.)

8. Subclasses + blueprints

dimos run teleop-hosted-xarm7                            # arm teleop (legacy module)
dimos run teleop-hosted-go2                              # mobile-base teleop (legacy module)
dimos run teleop-hosted-go2-transport                    # Go2 over CF *transports* (preferred)
dimos run teleop-hosted-go2-multicam                     # + RealSense mux'd into the track
# record any variant by composing the generic recorder:
dimos run teleop-hosted-go2-transport teleop-recorder

Run the transport variants with the broker key: TRANSPORTS__BROKER__API_KEY=dtk_live_... (was TELEOP_API_KEY before #2499) or -o transports.broker.api_key=....

Blueprint	Path	Notes
`teleop-hosted-xarm7`	legacy module	`HostedArmTeleopModule` → coordinator IK
`teleop-hosted-go2`	legacy module	`HostedTwistTeleopModule`
`teleop-hosted-go2-transport`	transport	`Go2HostedConnection`, one CF session, robot commands + telemetry
`teleop-hosted-go2-multicam`	transport	adds RealSense (`enable_depth=False`) mux'd into the video track

The dedicated hosted-teleop-recorder blueprint was removed — it only set a recordings path. Use the generic teleop-recorder composed onto any variant.

Operator-facing streams (controller/joy/twist/commands) ride WebRTC via the broker. Robot-internal streams (coordinator commands, recorder inputs, the RealSense → mux hop) stay on LCM.

Synced from DIM-981 by summer

Hosted Teleop Overview

Description