OpenwaterHealth / OpenwaterHealth/openmotion-sdk

Dropped scheduled dark is typed "light", silently doubling the dark-correction interval

Open
#175 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
3
Forks
5
Avg merge
12h 12m
Merged PRs (30d)
12

Description

Split out of #114 (item 4), which was deliberately scoped to the NaN-laundering fix only (PR #174).

Problem

timestamp_repair.py:341 hardcodes "frame_type": "light" for every synthetic NaN-fill row it inserts for a frame-id gap. If the dropped frame was a scheduled dark (abs_id == discard_count + 1, or (abs_id - 1) % dark_interval == 0 — see classify.py:_is_dark), the fill row enters PendingInterval as a NaN light sample. The dark boundary that should have closed the interval never arrives, so the interval silently spans to the next scheduled dark — the dark-correction interval doubles, with no log and no diagnostic event.

Since PR #174 the corrupted fill row at least emits as NaN rather than as a finite BFI, so the symptom is now missing data instead of manufactured data. But the doubled interval is still silent: every light frame in the widened interval gets its dark baseline linearly interpolated across twice the intended span, and nothing tells the operator or the analyst.

Why it wasn't fixed with #114

Typing the fill row "dark" on its own makes things worse. The row's u1/std are NaN, so:

  • DarkHistory.append stores a NaN DarkObservation -> HybridRealtimePredictor produces NaN baselines for the realtime path until the ring buffer rotates it out.
  • PendingInterval.set_right_dark / set_left_dark take it as an interval boundary -> LinearInterpolation gets d_prev.u1 or d_next.u1 NaN -> every frame in both adjacent intervals corrects to NaN.

That trades a silent doubled interval for two entirely NaN intervals plus a poisoned realtime predictor.

Proposed work

  1. dark.py — reject non-finite dark observations: don't append them to DarkHistory, don't accept them as an interval boundary. Decide and document the fallback (most likely: treat as a missed dark, keep the interval open to the next real dark, but emit an explicit event).
  2. Add a diagnostic event + WARNING for a missed dark, so a doubled interval is visible in the record rather than inferred.
  3. timestamp_repair.py — classify a synthetic fill row as "dark" when its abs id lands on the dark schedule, so the missed dark is explicit rather than mislabeled.
  4. Consider surfacing the count of missed darks in the scan summary / session_meta.

Needs 1 before 3, or the poisoning above.

Tests

  • Dropped scheduled dark -> fill row typed dark, missed-dark event emitted, interval handling explicit, no NaN corrected frames leaking from the boundary.
  • Non-finite dark observation never reaches DarkHistory or an interval boundary.
  • Realtime predictor unaffected by a dropped dark.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with timestamp_repair.py:341 and classify.py:_is_dark, then trace dark.py through DarkHistory, PendingInterval, HybridRealtimePredictor, and LinearInterpolation. Use the listed dropped-dark, non-finite-observation, and realtime-predictor scenarios as focused tests; done means the missed dark is diagnosed without NaN-corrected frames or predictor poisoning.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.