HarperFast / HarperFast/harper
NaN in @indexed Float sorts positive → leaks into > N range queries (silent-wrong); search_by_value(NaN) → 500
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
IEEE-754 `NaN` is mishandled by the `@indexed Float` path. NaN's stored bit pattern sorts into the **positive region** of the ordered index, so it leaks into `> 0` / `> N` range-query results (silent wrong results — IEEE-754 mandates every ordered comparison with NaN is false, so NaN must never appear in a range result). Separately, `search_by_value(val = NaN)` crashes with a 500.
**Severity: Medium** — silent wrong results in range queries. **Reachability is narrow:** NaN/±Infinity are not representable in JSON, so they can only enter via CBOR (`application/cbor`) or msgpack write paths — real for sensor/scientific apps where a computed reading can be `0/0`, `log(-x)`, overflow→Inf, etc.
## Symptoms
**Range-query leak (the real concern):** `search_by_conditions(val > 0)` and `(val > N)` on an `@indexed Float` column return rows whose value is NaN. `val < 0` correctly excludes NaN — confirming this is a sort-*position* bug (NaN's canonical encoding is a large positive integer under raw bit-reinterpretation), not random.
**Crash:** `search_by_value(val = NaN)` → HTTP 500, `TypeError: Cannot read properties of null (reading 'includes')` at `dist/dataLayer/harperBridge/ResourceBridge.js:393`. Returning `[]` would be the correct IEEE outcome; crashing is not.
**Sort-position inconsistency:** REST `?sort(fval)` ASC puts NaN last, DESC puts NaN first; SQL `ORDER BY fval` puts NaN between 0 and 1.0. No position is IEEE-correct (NaN is unordered) and REST and SQL disagree → silent-wrong for any sort-dependent query. Finite ordering (`-∞ < -1 < +1 < +∞`) is correct on both.
**GROUP BY merges NaN with NULL:** SQL `GROUP BY fval` yields `{fval: null, cnt: 1}` for the NaN row — NaN coerced to the null group key, no separate NaN group. A `WHERE fval IS NULL` query would then pick up the NaN row (identity confusion).
## Contained (good — no structural corruption)
CBOR re-encode round-trip (GET → re-PUT → GET) preserves NaN; neighbor-index integrity holds — a NaN DELETE fully cleans the index, and `> 0` / `< 0` / `= 1.0` all return correct non-NaN neighbors afterward (no wedge), both engines identical. The index *structure* is not corrupted — the defect is NaN's bad sort-position + NULL-merge, not a broader integrity failure.
## Not bugs (defensible companions)
JSON `Accept` emits `null` for NaN/±Inf (only spec-valid choice); `-0` stored as `+0`; SQL MIN/MAX→null and AVG/SUM→`{}` when a NaN/Inf is in the column (the finite-only subset aggregates correctly).
## Suggested fix
Normalize NaN to sort consistently (or exclude it from ordered-index range results), and guard the `search_by_value` NaN path to return `[]` (or a clean 4xx) instead of 500. One fix to the index ordering + a search-value guard addresses the whole cluster (range-leak, crash, sort-position, GROUP-BY-NULL-merge), which all flow from "NaN has no consistent sort position and isn't special-cased."
> Note when reproducing: drive it via the `search_by_value(val=NaN)` op + `search_by_conditions(val > 0)` specifically (REST FIQL `?fv=NaN` correctly returns 400 and does not hit this path).
---
*Found via exploratory QA (scenarios QA-207 / QA-209) against `7aaa5a152` on a feature branch; expected to reproduce on `main`. Filed by Claude (Opus 4.8) on Kris's go-ahead.*
Contributor guide
Research direction
Reproduce the issue through the search_by_value(val=NaN) and search_by_conditions(val > 0) operations, using QA scenarios QA-207 and QA-209. Inspect dist/dataLayer/harperBridge/ResourceBridge.js around line 393, then trace the indexed Float ordering and SQL grouping paths. Done means NaN no longer leaks into ranges, search_by_value does not return 500, and the reported sort and GROUP BY inconsistencies are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100