HarperFast / HarperFast/harper

SQL IN-list fans out one concurrent lookup per value with no cap — 2000-value list self-blocks ~10s (SQLSearch.ts)

Open
#2,393 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## What happens

A SQL `IN` list fans out **one concurrent lookup per value with no cap**: an N-value list issues N simultaneous `getDataByValue` calls. Measured: a 2000-value list self-blocks for 9.7s. Unlike the #1930 ReDoS family this path yields (it is not a worker-starvation vector — explicitly should NOT be bundled into the #1930 fix), but the uncapped fan-out is a self-inflicted thundering herd against the storage layer, and it scales with attacker- or app-controlled list size.

## Mechanism (on `origin/main` @ e16d9616)

`dataLayer/SQLSearch.ts:675`:

```ts
Promise.all(Array.from(...).map(async value => ... getDataByValue(...)))
```

byte-identical to when found. No shared concurrency-cap utility exists in the codebase to reuse. Reachability: the site is legacy-engine-only, but it is hit **by default** — `sql.engine: 'auto'` routes any query v2 can't handle to the legacy path, and v2 supports `IN` natively (`normalizer.ts`), so exposure = legacy-pinned configs plus every auto-fallback query carrying an IN clause.

## Existing work checked

harper#2240 closed via PR #2242 which only touched `resources/analytics/write.ts` (confirmed via the PR's file list) — unrelated, the retest flag on this finding is cleared. No open/closed issue covers IN-list fan-out or a lookup concurrency cap. #1930 is the adjacent family and deliberately excluded (different resource, different fix).

## What this does not prove

Whether the 9.7s self-block degrades *other* requests (the path yields; contention measured only against itself); RocksDB-only measurement.

---
*From dispatch QA finding F-243, verified against harper origin/main e16d9616, 2026-08-29.*

Contributor guide

Open the contributing guide

Research direction

Start in dataLayer/SQLSearch.ts around line 675 and trace the IN-list path through getDataByValue; review normalizer.ts for the v2 handling and the legacy fallback conditions. Determine a bounded lookup strategy and an appropriate cap, then verify that IN queries still return correct results without the uncapped fan-out and that the legacy path remains distinct from the #1930 fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, sql, typescript
Domain
backend, databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.