HarperFast / HarperFast/harper

SQL engine: silent-wrong results (offset-past-end, date literals, ROW_NUMBER) + unsupported constructs 500 not 400

Open
#1,397 3 comments 0 reactions 0 assignees View on GitHub
area:sql bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Three related SQL-engine correctness/observability defects, aggregated into one issue (the SQL engine is under active rewrite). All are **silent-wrong-result** or **wrong-status** behaviors in the AlaSQL-backed two-pass executor (`dataLayer/SQLSearch.ts`). All reproduce on both storage engines.

### A. `LIMIT N OFFSET M` returns the ENTIRE table when `OFFSET ≥ rowcount` (silent-wrong + over-fetch)

A query with an out-of-range offset returns **all rows** (LIMIT silently dropped) instead of an empty page. Verified: offsets 5000/5250/6000/10000 on a 5000-row table all returned 5000 rows; in-range offsets are correct and disjoint.

```
SELECT * FROM data.T ORDER BY k LIMIT 50 OFFSET 1000000 -- on a small table -> returns the WHOLE table
```

- **Impact:** breaks the universal "page until a short/empty result, then stop" idiom (a naive loop never terminates / reprocesses everything), and over-fetches the whole table into a response a client capped at N.
- **Root cause:** the two-pass executor applies LIMIT/OFFSET in the first pass, then `_finalSQL` (≈L1206-1217) *deletes* `statement.limit`/`statement.offset` so they aren't re-applied in the merge pass. When the first pass matches nothing (offset past end), the final pass runs over `__mergedData` with no cap and re-emits everything.
- **Fix:** preserve an empty result when the first pass yields zero rows due to an out-of-range offset (don't drop LIMIT on the re-emit), or re-apply LIMIT/OFFSET in the merge pass.

### B. Date-column filtering by a literal silently returns `[]`

On a `Date`-typed column, SQL filtering by a date literal silently returns the empty set:

```
SELECT * FROM data.T WHERE when = '2026-06-18T12:00:00.000Z' -- -> [] (rows exist)
SELECT * FROM data.T WHERE when < '2026-06-18T12:00:00.000Z' -- -> [] (string literal)
SELECT * FROM data.T WHERE when < 1750000000000 -- -> rows (numeric epoch works)
```

Only **numeric-epoch** bounds match; ISO-string and exact-epoch equality both yield `[]`. REST FIQL (`?when=ge=date:`) filters correctly, and ORDER BY on the column is correctly chronological — so the data/index are fine; it's the SQL literal→Date comparison path. A SQL user querying a date column by the obvious literal believes "no matching rows" when rows exist.
- **Fix:** coerce SQL date literals (ISO string / epoch) to the stored Date representation before matching, or reject an un-coercible date predicate loudly.

### C. `ROW_NUMBER() OVER (...)` silently ignores its OVER clause → wrong rankings; unsupported constructs return 500 not 400

```
SELECT name, ROW_NUMBER() OVER (PARTITION BY deptId ORDER BY salary DESC) AS rn
FROM data.Employee WHERE deptId=1
-- -> 200, but ranks assigned in physical/id-scan order, ignoring the OVER ORDER BY (silently wrong)
```

The column classifier keys only on `aggregatorid`/`funcid` (`SQLSearch.ts` ~L372-402), so `ROW_NUMBER` is treated as a plain scalar and its PARTITION/ORDER BY is dropped.

Companion observability gap (same engine): of 13 advanced constructs probed, only `DISTINCT` and `HAVING` are supported; the other 10 (correlated/uncorrelated subqueries, `IN`/`NOT IN (SELECT)`, `UNION`/`UNION ALL`, `RANK`, `CASE WHEN`, derived table in FROM) return **HTTP 500** with internal-sounding messages (`Circular reference`, `unknown column …`, `schema not defined for table undefined`) instead of a clean 400 "construct not supported". Source: bare-string / `new Error(SEARCH_ERROR_MSG)` throws (`SQLSearch.ts` L123-171/480-484, `SelectValidator.ts` L87/252/311) → ops API maps to 500. (Same systemic 4xx-as-500 pattern noted in #1299.) `CASE WHEN` being unsupported is itself notable.
- **Fix:** reject window functions loudly (400) until OVER is honored (or honor it); map unsupported-construct/validator failures to 400 with a clear message.

## Scope note (clean, for contrast)

JOINs (inner/left-null-fill/self/3-way/aggregate), numeric range/BETWEEN boundaries (Int/Long/Float, inclusive/exclusive, reversed→empty), and GROUP BY aggregates are all **correct** against hand-computed truth on both engines — the silent-wrong issues are specifically: out-of-range offset, date literals, and window functions.

---
*Surfaced by the QA-explorer campaign against Harper `7aaa5a152`. `dataLayer/SQLSearch.ts` is byte-identical to `main` @`6797f091d`, so all three reproduce on main. Ready-to-promote regression tests exist for each. Filed by Claude (Opus 4.8) for @kris.*

Contributor guide

Open the contributing guide

Research direction

Start with dataLayer/SQLSearch.ts, especially the _finalSQL limit/offset handling and the column-classifier paths, then inspect SelectValidator.ts and the existing ready-to-promote regression tests. Verify out-of-range offsets remain empty, date literals match Date columns, ROW_NUMBER does not silently ignore OVER, and unsupported constructs produce clear 400 responses rather than 500s.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql, typescript
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.