feat: Real-Time Dense Point Cloud — Camera + WiFi CSI + mmWave Sensor Fusion

Introduction

RuView now supports real-time dense 3D point cloud generation by fusing multiple sensor modalities: camera depth, WiFi CSI, and mmWave radar. This is the first WiFi sensing platform that combines through-wall radio sensing with visual depth estimation into a unified spatial model.

The system runs entirely locally — no cloud, no external APIs — using the RTX 5080 GPU for neural depth estimation and ESP32 nodes for WiFi CSI sensing.

Features

7-Component Sensor Fusion Pipeline (all Rust)

#	Component	What it does	Input	Output
1	ADR-018 Parser	Decodes ESP32 CSI binary frames	UDP packets	I/Q subcarrier amplitudes + phases
2	WiFlow Pose	17 COCO keypoints from WiFi CSI	35 subcarriers × 20 timesteps	Skeleton coordinates
3	Camera Depth	MiDaS monocular depth on GPU	640×480 RGB	Depth map → 3D points
4	Sensor Fusion	Merge camera + CSI into unified cloud	Multiple point clouds	Fused point cloud + Gaussian splats
5	RF Tomography	Through-wall occupancy from CSI	Per-node RSSI	8×8×4 voxel grid
6	Vital Signs	Breathing rate from CSI phase	Phase time series	BPM estimate
7	Motion-Adaptive	Skip expensive depth when idle	CSI motion score	Adaptive capture rate

Brain Integration

Spatial observations are stored in the ruOS brain every 60 seconds:

• spatial-observation — room scan summary (camera frames, CSI frames, motion, vitals)
• spatial-motion — motion events when score > 30%
• spatial-vitals — breathing rate + motion percentage

The ruOS agent can search the brain for spatial context: "is anyone in the room?", "when was motion last detected?"

Capabilities Comparison

Capability	RuView (before)	RuView (with point cloud)
Sensing modality	WiFi CSI only	Camera + WiFi CSI + mmWave
Output format	2D pose keypoints	3D point cloud + Gaussian splats
Depth estimation	None	MiDaS neural depth (GPU)
Visualization	2D skeleton overlay	Interactive 3D Three.js viewer
Room modeling	None	40K+ voxel room model from 20 frames
Brain integration	CSI logs only	Spatial observations + vitals + motion
Motion detection	CSI amplitude threshold	CSI variance + adaptive capture rate
Output files	JSON/CSV	PLY, Gaussian splats JSON, brain memories
Real-time streaming	WebSocket (2D)	HTTP polling + Three.js 3D (2 fps)

User Guide

Prerequisites

• Linux with USB camera (/dev/video0)
• ESP32-S3 with RuView CSI firmware (optional, for WiFi sensing)
• Rust 1.77+ (for building from source)

Quick Start

# Build
cd rust-port/wifi-densepose-rs
cargo build --release -p wifi-densepose-pointcloud

# Run demo (no hardware needed)
./target/release/ruview-pointcloud demo
# → Creates demo_pointcloud.ply (40K points) + demo_splats.json

# Start live server with camera
./target/release/ruview-pointcloud serve --port 9880
# → Open http://localhost:9880 for interactive 3D viewer

Practical Examples

Example 1: Quick room scan

# Capture 1 frame and save as PLY
ruview-pointcloud capture --output my_room.ply
# Open in MeshLab, CloudCompare, or any PLY viewer

Example 2: Live monitoring with CSI

# Provision ESP32 to send CSI to this machine
python3 firmware/esp32-csi-node/provision.py \
    --port /dev/ttyACM0 \
    --ssid "MyWiFi" --password "MyPassword" \
    --target-ip $(hostname -I | awk '{print $1}') --target-port 3333

# Start server — auto-detects camera + CSI
ruview-pointcloud serve --port 9880
# Info panel shows: motion %, breathing BPM, skeleton status

Example 3: Depth calibration training

# Place reference objects at known distances, then:
ruview-pointcloud train --data-dir ./my-calibration
# Outputs: calibration.json, preference_pairs.jsonl, brain memories

Example 4: Brain-aware room monitoring

# Server syncs to brain every 60s automatically
# Search brain for spatial context:
curl -s -X POST http://127.0.0.1:9876/brain/search \
    -H "Content-Type: application/json" \
    -d '{"query":"motion detected in room","k":3}'

API Endpoints

Endpoint	Description
GET /	Interactive Three.js 3D point cloud viewer
GET /health	Health check
GET /api/status	Full pipeline status (camera, CSI, vitals, motion)
GET /api/cloud	Point cloud data (up to 1000 points)
GET /api/splats	Gaussian splats for rendering

Performance

Metric	Value
Pipeline latency	22ms
API throughput	905 req/s
Room model quality	40,110 voxels (5cm resolution, 20 frames)
Gaussian splats	2,000+ per frame
CSI throughput	112K+ frames processed
Binary size	1.0 MB (arm64), ~2 MB (x86_64)

Related

• PR: #405
• ADR-SYS-0021: Real-time dense point cloud architecture
• WiFlow model: v0.7.0 (92.9% PCK@20, 186K params)
• Firmware: v0.6.1-esp32 (ADR-018 binary CSI output)

Files

New crate: rust-port/wifi-densepose-rs/crates/wifi-densepose-pointcloud/

• 10 Rust modules, ~3,500 LOC
• Binary: ruview-pointcloud
• Dependencies: axum, reqwest, chrono, clap, serde_json

Start by reviewing PR #405 and the new crate at rust-port/wifi-densepose-rs/crates/wifi-densepose-pointcloud/, then run the documented cargo build and demo commands. Done would mean the point-cloud binary, live server endpoints, sensor-fusion pipeline, and documented outputs work as described.