OpenwaterHealth / OpenwaterHealth/openmotion-sdk
Dropped scheduled dark is typed "light", silently doubling the dark-correction interval
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3
- Forks
- 5
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 12
Description
Split out of #114 (item 4), which was deliberately scoped to the NaN-laundering fix only (PR #174).
Problem
timestamp_repair.py:341 hardcodes "frame_type": "light" for every synthetic NaN-fill row it inserts for a frame-id gap. If the dropped frame was a scheduled dark (abs_id == discard_count + 1, or (abs_id - 1) % dark_interval == 0 — see classify.py:_is_dark), the fill row enters PendingInterval as a NaN light sample. The dark boundary that should have closed the interval never arrives, so the interval silently spans to the next scheduled dark — the dark-correction interval doubles, with no log and no diagnostic event.
Since PR #174 the corrupted fill row at least emits as NaN rather than as a finite BFI, so the symptom is now missing data instead of manufactured data. But the doubled interval is still silent: every light frame in the widened interval gets its dark baseline linearly interpolated across twice the intended span, and nothing tells the operator or the analyst.
Why it wasn't fixed with #114
Typing the fill row "dark" on its own makes things worse. The row's u1/std are NaN, so:
DarkHistory.appendstores a NaNDarkObservation->HybridRealtimePredictorproduces NaN baselines for the realtime path until the ring buffer rotates it out.PendingInterval.set_right_dark/set_left_darktake it as an interval boundary ->LinearInterpolationgetsd_prev.u1ord_next.u1NaN -> every frame in both adjacent intervals corrects to NaN.
That trades a silent doubled interval for two entirely NaN intervals plus a poisoned realtime predictor.
Proposed work
dark.py— reject non-finite dark observations: don't append them toDarkHistory, don't accept them as an interval boundary. Decide and document the fallback (most likely: treat as a missed dark, keep the interval open to the next real dark, but emit an explicit event).- Add a diagnostic event + WARNING for a missed dark, so a doubled interval is visible in the record rather than inferred.
timestamp_repair.py— classify a synthetic fill row as"dark"when its abs id lands on the dark schedule, so the missed dark is explicit rather than mislabeled.- Consider surfacing the count of missed darks in the scan summary /
session_meta.
Needs 1 before 3, or the poisoning above.
Tests
- Dropped scheduled dark -> fill row typed
dark, missed-dark event emitted, interval handling explicit, no NaN corrected frames leaking from the boundary. - Non-finite dark observation never reaches
DarkHistoryor an interval boundary. - Realtime predictor unaffected by a dropped dark.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with timestamp_repair.py:341 and classify.py:_is_dark, then trace dark.py through DarkHistory, PendingInterval, HybridRealtimePredictor, and LinearInterpolation. Use the listed dropped-dark, non-finite-observation, and realtime-predictor scenarios as focused tests; done means the missed dark is diagnosed without NaN-corrected frames or predictor poisoning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100