OpenwaterHealth / OpenwaterHealth/openmotion-sdk

Host-side sink stalls starve the USB drain and overflow the firmware histo queue (frame drops on long scans)

Open
#116 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3
Forks
5
Avg merge
12h 12m
Merged PRs (30d)
12

Description

Problem

On the 2026-06 long-scan soak the sensor firmware logged ~833 HISTO enqueue fail: queue full events (enq-deq=4, txfail=0). Root cause is host-side: the ScanRunner consume loop runs on the same thread that drains the live USB histogram batch queue, so a slow disk/DB sink (SQLite commit/fsync, CSV flush) stalls the drain → backpressures the parser → dev.read → the firmware's deliberately shallow 4-deep (~100 ms) histo queue overflows and frames are dropped. Brief (>100 ms, <1 s) host stalls — not a USB fault, not the persistent histo-wedge.

Fix

  • AsyncSink: run storage sinks' consume() on a dedicated worker thread behind a bounded buffer, so transient sink stalls never touch the USB path (sustained overload still backpressures — correct).
  • Deepen LiveUsbSource._batch_queue to an explicit ~3 s budget computed at the true per-camera row rate (CAPTURE_HZ × active_cameras).
  • Instrument StreamInterface packet-queue puts so sub-second host stalls and drops are counted, warned, and summarised per scan (previously invisible below 1 s).
  • Document the sink consume() latency budget and shared-payload contract in docs/API.md / the Sink protocol.

Out of scope: bumping firmware HISTO_QUEUE_SIZE (RAM-bound on the sensor; host-side cause removed instead).

PRs

  • #96 — original implementation (superseded; close when #113 merges)
  • #113 — revised: fixes the ~16× batch-queue sizing error, the AsyncSink join-timeout finalize race (could delete a DB session as "empty"), stale-queue reuse poisoning, and warn-token cross-suppression; rebased onto next through #111/#108

Verification

  • tests/test_pipeline/, tests/test_scan_workflow.py, tests/test_stream_interface_backpressure.py: 274 passed / 0 failed on the rebased branch.
  • HIL follow-up: repeat a long-scan soak and confirm zero HISTO enqueue fail events and a clean backpressure summary in the stop_streaming log line.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing revised PR #113 and the AsyncSink, LiveUsbSource._batch_queue, and StreamInterface entry points named in the issue. Run the listed pipeline, scan-workflow, and backpressure tests, then repeat the long-scan HIL soak; done means the tests pass, no HISTO enqueue failures occur, and stop_streaming reports clean backpressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, embedded-iot, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.