HarperFast / HarperFast/harper

NaN in @indexed Float sorts positive → leaks into > N range queries (silent-wrong); search_by_value(NaN) → 500

Open
#1,436 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

IEEE-754 `NaN` is mishandled by the `@indexed Float` path. NaN's stored bit pattern sorts into the **positive region** of the ordered index, so it leaks into `> 0` / `> N` range-query results (silent wrong results — IEEE-754 mandates every ordered comparison with NaN is false, so NaN must never appear in a range result). Separately, `search_by_value(val = NaN)` crashes with a 500.

**Severity: Medium** — silent wrong results in range queries. **Reachability is narrow:** NaN/±Infinity are not representable in JSON, so they can only enter via CBOR (`application/cbor`) or msgpack write paths — real for sensor/scientific apps where a computed reading can be `0/0`, `log(-x)`, overflow→Inf, etc.

## Symptoms

**Range-query leak (the real concern):** `search_by_conditions(val > 0)` and `(val > N)` on an `@indexed Float` column return rows whose value is NaN. `val < 0` correctly excludes NaN — confirming this is a sort-*position* bug (NaN's canonical encoding is a large positive integer under raw bit-reinterpretation), not random.

**Crash:** `search_by_value(val = NaN)` → HTTP 500, `TypeError: Cannot read properties of null (reading 'includes')` at `dist/dataLayer/harperBridge/ResourceBridge.js:393`. Returning `[]` would be the correct IEEE outcome; crashing is not.

**Sort-position inconsistency:** REST `?sort(fval)` ASC puts NaN last, DESC puts NaN first; SQL `ORDER BY fval` puts NaN between 0 and 1.0. No position is IEEE-correct (NaN is unordered) and REST and SQL disagree → silent-wrong for any sort-dependent query. Finite ordering (`-∞ < -1 < +1 < +∞`) is correct on both.

**GROUP BY merges NaN with NULL:** SQL `GROUP BY fval` yields `{fval: null, cnt: 1}` for the NaN row — NaN coerced to the null group key, no separate NaN group. A `WHERE fval IS NULL` query would then pick up the NaN row (identity confusion).

## Contained (good — no structural corruption)

CBOR re-encode round-trip (GET → re-PUT → GET) preserves NaN; neighbor-index integrity holds — a NaN DELETE fully cleans the index, and `> 0` / `< 0` / `= 1.0` all return correct non-NaN neighbors afterward (no wedge), both engines identical. The index *structure* is not corrupted — the defect is NaN's bad sort-position + NULL-merge, not a broader integrity failure.

## Not bugs (defensible companions)

JSON `Accept` emits `null` for NaN/±Inf (only spec-valid choice); `-0` stored as `+0`; SQL MIN/MAX→null and AVG/SUM→`{}` when a NaN/Inf is in the column (the finite-only subset aggregates correctly).

## Suggested fix

Normalize NaN to sort consistently (or exclude it from ordered-index range results), and guard the `search_by_value` NaN path to return `[]` (or a clean 4xx) instead of 500. One fix to the index ordering + a search-value guard addresses the whole cluster (range-leak, crash, sort-position, GROUP-BY-NULL-merge), which all flow from "NaN has no consistent sort position and isn't special-cased."

> Note when reproducing: drive it via the `search_by_value(val=NaN)` op + `search_by_conditions(val > 0)` specifically (REST FIQL `?fv=NaN` correctly returns 400 and does not hit this path).

---
*Found via exploratory QA (scenarios QA-207 / QA-209) against `7aaa5a152` on a feature branch; expected to reproduce on `main`. Filed by Claude (Opus 4.8) on Kris's go-ahead.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue through the search_by_value(val=NaN) and search_by_conditions(val > 0) operations, using QA scenarios QA-207 and QA-209. Inspect dist/dataLayer/harperBridge/ResourceBridge.js around line 393, then trace the indexed Float ordering and SQL grouping paths. Done means NaN no longer leaks into ranges, search_by_value does not return 500, and the reported sort and GROUP BY inconsistencies are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.