sensing-server: processing loop silently stops with >1 ESP32 node — /health tick freezes while HTTP keeps serving (stale readings, calibration lost)

Summary

With more than one ESP32 node streaming CSI, the sensing server's processing loop stops permanently after a few minutes. The HTTP server keeps serving, so the failure is silent: /health returns status: ok with a frozen tick, /api/v1/nodes keeps returning the last values it saw, and any reading taken during this window looks plausible but is stale. Nothing is logged — no panic, no error.

The nodes are unaffected and keep transmitting throughout.

Environment

Server	ruvnet/wifi-densepose:latest (sha-b370d4f, arm64), Docker Desktop on macOS 15
Args	--source esp32, UDP 5005, RUVIEW_UDP_BIND=0.0.0.0 with a source allowlist
Nodes	3 × ESP32-S3-DevKitC-1 N16R8, firmware 0.8.8, edge_tier=2, TDM slots 0/1/2 of 3
Rate	~30 CSI frames/s per node

Symptoms when frozen

$ curl -s localhost:3000/health          # unchanged across minutes
{"clients":0,"source":"esp32:offline","status":"ok","tick":27560}
$ curl -s localhost:3000/api/v1/nodes    # csi_sequence frozen, all nodes "stale"

• Container memory drops from ~19.6 MiB to ~4.3 MiB; CPU ~0.1%.
• No log output at all after the freeze; the last lines are normal startup or WebSocket connect/disconnect entries.
• /api/v1/status reports source_state: disconnected, engine_error_count: 0.

The nodes are still sending

Stopping the container and binding UDP 5005 directly on the host shows all three nodes transmitting normally:

packets per source in 10s: {'192.168.1.190': 365, '192.168.1.101': 374, '192.168.1.98': 373}

So this is server-side, not a node or network fault.

Measurements (node count is the variable)

Nodes streaming	Result
1	stable ≥10 min, repeatedly; a full 606 s calibration ran to completion
2 (both n_subcarriers=64)	froze after ~5 min
3	froze 4 separate times, typically within 10–15 min

To isolate a single node I retargeted the others to an unused UDP port rather than filtering by address, because Docker Desktop rewrites the source address before the container sees it (a per-node RUVIEW_UDP_ALLOW entry therefore matches nothing).

Partial mitigation

Setting WDP_TDM_SLOTS=3 and WDP_TDM_SLOT_US=20000 (fusion guard 72000 µs hard / 48000 µs soft, up from 60000/20000) let three nodes run a clean 10-minute window. It is not a cure: a freeze still occurred about 13 minutes later.

Possibly related: heterogeneous and drifting CSI grids

POST /api/v1/calibration/start?source_node_id=N reports different bound grids per node at the same moment:

• node 1 → {"n_subcarriers": 128, "ppdu_type": 0}
• node 2 → {"n_subcarriers": 64, "ppdu_type": 0}
• node 3 → {"n_subcarriers": 64, "ppdu_type": 0}

and the same node's grid changes over time (node 1 was 64 earlier in the session). This mixed-width case is what CHANGELOG entry for resample_to_canonical describes, which is why I suspect the multistatic fusion path — though the freeze also occurs with two nodes reporting the same width, so grid mismatch alone does not explain it.

Impact

Beyond the stale readings, a freeze destroys calibration. /calibration/stop fails with calibration_grid_stale if the loop dies mid-run, and because calibration is in-memory only (raw_calibration_frames_persisted: false), restarting to recover loses a stored baseline entirely. Two 10-minute empty-room calibrations were lost this way.

Workaround in use

An external watchdog polls /health and restarts the container when tick stops advancing. It fires often enough to be necessary, which also means an unattended deployment silently loses calibration on a schedule.

Happy to run further diagnostics on this hardware if that helps narrow it down.

Start with the sensing-server processing loop, UDP ingest, the /health tick, and the multistatic fusion path described in the report. Reproduce with two or more ESP32 nodes while monitoring tick advancement, source state, and engine errors; done means the loop continues processing, failures are visible in logs, and readings and calibration remain current.