HarperFast / HarperFast/harper
REST chained range conditions (`a=ge=X&=le=Y`) never coerce the chained leg's value — silent superset or empty results on numeric attributes
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A REST chained range condition (`attr=ge=X&=le=Y`) never type-coerces the chained leg's value. The planner collapses the chain into a single range comparator with a **mixed-type bound pair** — e.g. `[175, "180"]` — and since numbers sort below strings in key order, the string bound degenerates: the query silently returns a **superset** (string upper bound never excludes anything) or an **empty set** (string lower bound excludes everything), depending on which leg was chained. No error is raised, and the indexed and unindexed execution paths return the same wrong answer.
The chained syntax itself is semantically correct: once both values carry the right type (programmatic conditions, or `number:`-prefixed REST values), the collapse yields exactly the element-scoped range matching that the RQL 2.0 specification formalizes for chains — verified on both the indexed and unindexed paths. Only the REST value-typing path is broken.
## Reproduction (synthetic, unit level)
Table with `age: Int` (scalar) and `lengths: { elements: { type: Int } }` (array), each in an indexed and an unindexed variant; records:
| id | age | lengths |
|---|---|---|
| 1 | 172 | [172, 174, 181] |
| 2 | 176 | [176, 178] |
| 3 | 150 | [150] |
| 4 | 175 | [175] |
| 5 | 181 | null |
Queries via `parseQuery(...)` → `Table.search({ conditions, allowFullScan: true })` (same route the REST layer takes):
| query | observed | expected |
|---|---|---|
| `age=ge=175&=le=180` | **[2,4,5]** | [2,4] — id 5 (age 181) must not match `le=180` |
| `age=le=180&=ge=175` (legs swapped) | **[]** | [2,4] |
| `lengths=ge=175&=le=180` (unindexed array) | **[1,2,4]** | [2,4] — id 1 has no single element in [175,180] |
| `age=ge=number:175&=le=number:180` (typed prefix) | [2,4] ✓ | [2,4] |
| programmatic `{comparator:'ge', value:175, chainedConditions:[{comparator:'le', value:180}]}` | [2,4] ✓ | [2,4] |
Indexed variants of the broken queries return the same wrong sets (plus the separate duplicate-row defect for multi-value attributes, filed separately).
## Analysis
1. The REST parser decodes FIQL values with `typedDecoding` (`resources/search.ts:1593`), which returns a **string** unless the value carries an explicit `number:`/`boolean:`/`date:` prefix. So both legs of `age=ge=175&=le=180` arrive as strings.
2. `prepareConditions` (`resources/Table.ts:3496-3497`) coerces `condition.value` to the attribute type (or auto-coerces for `COERCIBLE_OPERATORS`) — but it never touches `condition.chainedConditions[i].value`. The chained-leg loop right below (`resources/Table.ts:3500-3529`) then collapses the chain into `condition.value = [lower.value, upper.value]` with the chained leg's value still a raw string.
3. Execution compares with key ordering (`compareKeys`, and the index range scan), where every number sorts below every string: a string upper bound is `> ` all numbers (no-op), a string lower bound is `>` all numbers (excludes all).
Fix sketch: coerce chained-leg values in the same place `condition.value` is coerced (or resolve the coercion inside the collapse before building the pair).
## Severity
Silent wrong result set on the documented REST range-chaining syntax, for any non-string attribute, on both the indexed and unindexed paths. Verified on `main` (a21910cff); the identical collapse/coercion code ships in v5.2.7. Workaround exists (`number:`-prefixed values) but nothing steers users to it — the untyped form fails silently.
Pinning/regression tests for the correct-typed behavior (and skipped tests asserting the correct behavior for this defect) are in the linked verification PR.
Contributor guide
Research direction
Start with typedDecoding in resources/search.ts:1593, then trace condition preparation and chained-condition collapse in resources/Table.ts:3496-3529 through parseQuery(...) and Table.search(...). Add regression coverage for numeric scalar and array ranges on indexed and unindexed paths; done means untyped chained REST values produce [2,4] in every listed broken case while typed-prefix and programmatic cases remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- api, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100