HarperFast / HarperFast/harper
SQL IN-list fans out one concurrent lookup per value with no cap — 2000-value list self-blocks ~10s (SQLSearch.ts)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## What happens
A SQL `IN` list fans out **one concurrent lookup per value with no cap**: an N-value list issues N simultaneous `getDataByValue` calls. Measured: a 2000-value list self-blocks for 9.7s. Unlike the #1930 ReDoS family this path yields (it is not a worker-starvation vector — explicitly should NOT be bundled into the #1930 fix), but the uncapped fan-out is a self-inflicted thundering herd against the storage layer, and it scales with attacker- or app-controlled list size.
## Mechanism (on `origin/main` @ e16d9616)
`dataLayer/SQLSearch.ts:675`:
```ts
Promise.all(Array.from(...).map(async value => ... getDataByValue(...)))
```
byte-identical to when found. No shared concurrency-cap utility exists in the codebase to reuse. Reachability: the site is legacy-engine-only, but it is hit **by default** — `sql.engine: 'auto'` routes any query v2 can't handle to the legacy path, and v2 supports `IN` natively (`normalizer.ts`), so exposure = legacy-pinned configs plus every auto-fallback query carrying an IN clause.
## Existing work checked
harper#2240 closed via PR #2242 which only touched `resources/analytics/write.ts` (confirmed via the PR's file list) — unrelated, the retest flag on this finding is cleared. No open/closed issue covers IN-list fan-out or a lookup concurrency cap. #1930 is the adjacent family and deliberately excluded (different resource, different fix).
## What this does not prove
Whether the 9.7s self-block degrades *other* requests (the path yields; contention measured only against itself); RocksDB-only measurement.
---
*From dispatch QA finding F-243, verified against harper origin/main e16d9616, 2026-08-29.*
Contributor guide
Research direction
Start in dataLayer/SQLSearch.ts around line 675 and trace the IN-list path through getDataByValue; review normalizer.ts for the v2 handling and the legacy fallback conditions. Determine a bounded lookup strategy and an appropriate cap, then verify that IN queries still return correct results without the uncapped fan-out and that the legacy path remains distinct from the #1930 fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, sql, typescript
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100