HarperFast / HarperFast/harper
SQL engine: silent-wrong results (offset-past-end, date literals, ROW_NUMBER) + unsupported constructs 500 not 400
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Three related SQL-engine correctness/observability defects, aggregated into one issue (the SQL engine is under active rewrite). All are **silent-wrong-result** or **wrong-status** behaviors in the AlaSQL-backed two-pass executor (`dataLayer/SQLSearch.ts`). All reproduce on both storage engines.
### A. `LIMIT N OFFSET M` returns the ENTIRE table when `OFFSET ≥ rowcount` (silent-wrong + over-fetch)
A query with an out-of-range offset returns **all rows** (LIMIT silently dropped) instead of an empty page. Verified: offsets 5000/5250/6000/10000 on a 5000-row table all returned 5000 rows; in-range offsets are correct and disjoint.
```
SELECT * FROM data.T ORDER BY k LIMIT 50 OFFSET 1000000 -- on a small table -> returns the WHOLE table
```
- **Impact:** breaks the universal "page until a short/empty result, then stop" idiom (a naive loop never terminates / reprocesses everything), and over-fetches the whole table into a response a client capped at N.
- **Root cause:** the two-pass executor applies LIMIT/OFFSET in the first pass, then `_finalSQL` (≈L1206-1217) *deletes* `statement.limit`/`statement.offset` so they aren't re-applied in the merge pass. When the first pass matches nothing (offset past end), the final pass runs over `__mergedData` with no cap and re-emits everything.
- **Fix:** preserve an empty result when the first pass yields zero rows due to an out-of-range offset (don't drop LIMIT on the re-emit), or re-apply LIMIT/OFFSET in the merge pass.
### B. Date-column filtering by a literal silently returns `[]`
On a `Date`-typed column, SQL filtering by a date literal silently returns the empty set:
```
SELECT * FROM data.T WHERE when = '2026-06-18T12:00:00.000Z' -- -> [] (rows exist)
SELECT * FROM data.T WHERE when < '2026-06-18T12:00:00.000Z' -- -> [] (string literal)
SELECT * FROM data.T WHERE when < 1750000000000 -- -> rows (numeric epoch works)
```
Only **numeric-epoch** bounds match; ISO-string and exact-epoch equality both yield `[]`. REST FIQL (`?when=ge=date:`) filters correctly, and ORDER BY on the column is correctly chronological — so the data/index are fine; it's the SQL literal→Date comparison path. A SQL user querying a date column by the obvious literal believes "no matching rows" when rows exist.
- **Fix:** coerce SQL date literals (ISO string / epoch) to the stored Date representation before matching, or reject an un-coercible date predicate loudly.
### C. `ROW_NUMBER() OVER (...)` silently ignores its OVER clause → wrong rankings; unsupported constructs return 500 not 400
```
SELECT name, ROW_NUMBER() OVER (PARTITION BY deptId ORDER BY salary DESC) AS rn
FROM data.Employee WHERE deptId=1
-- -> 200, but ranks assigned in physical/id-scan order, ignoring the OVER ORDER BY (silently wrong)
```
The column classifier keys only on `aggregatorid`/`funcid` (`SQLSearch.ts` ~L372-402), so `ROW_NUMBER` is treated as a plain scalar and its PARTITION/ORDER BY is dropped.
Companion observability gap (same engine): of 13 advanced constructs probed, only `DISTINCT` and `HAVING` are supported; the other 10 (correlated/uncorrelated subqueries, `IN`/`NOT IN (SELECT)`, `UNION`/`UNION ALL`, `RANK`, `CASE WHEN`, derived table in FROM) return **HTTP 500** with internal-sounding messages (`Circular reference`, `unknown column …`, `schema not defined for table undefined`) instead of a clean 400 "construct not supported". Source: bare-string / `new Error(SEARCH_ERROR_MSG)` throws (`SQLSearch.ts` L123-171/480-484, `SelectValidator.ts` L87/252/311) → ops API maps to 500. (Same systemic 4xx-as-500 pattern noted in #1299.) `CASE WHEN` being unsupported is itself notable.
- **Fix:** reject window functions loudly (400) until OVER is honored (or honor it); map unsupported-construct/validator failures to 400 with a clear message.
## Scope note (clean, for contrast)
JOINs (inner/left-null-fill/self/3-way/aggregate), numeric range/BETWEEN boundaries (Int/Long/Float, inclusive/exclusive, reversed→empty), and GROUP BY aggregates are all **correct** against hand-computed truth on both engines — the silent-wrong issues are specifically: out-of-range offset, date literals, and window functions.
---
*Surfaced by the QA-explorer campaign against Harper `7aaa5a152`. `dataLayer/SQLSearch.ts` is byte-identical to `main` @`6797f091d`, so all three reproduce on main. Ready-to-promote regression tests exist for each. Filed by Claude (Opus 4.8) for @kris.*
Contributor guide
Research direction
Start with dataLayer/SQLSearch.ts, especially the _finalSQL limit/offset handling and the column-classifier paths, then inspect SelectValidator.ts and the existing ready-to-promote regression tests. Verify out-of-range offsets remain empty, date literals match Date columns, ROW_NUMBER does not silently ignore OVER, and unsupported constructs produce clear 400 responses rather than 500s.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100