HarperFast / HarperFast/harper
getRecordCount on a large RocksDB table blocks the event loop for seconds: getKeysCount is one synchronous native full-key iteration (measured 2.47s on 2.2M keys; serve-path request went 6ms -> 1009ms)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`getRecordCount()` on a large RocksDB table blocks the calling thread's event loop for **seconds** —
one synchronous native call iterates every key in the table with no yields. Any HTTP request routed
to that worker during the call waits the full remaining duration. We measured this end-to-end in
production: a **2,471ms unbroken native block** on a healthy 2.2M-key table, and a cache-hit HEAD
request that normally serves in 6ms taking **1,009ms** because it landed on the blocked worker.
Composes badly with HarperFast/harper#2107: a leaked snapshot retains obsolete versions, so the same full-key
iteration also walks every dead version — the block grows with process age (we project 10s+ on a
table with 12M retained versions for ~1.6M live rows).
## Mechanism
`core/resources/Table.ts` `getRecordCount()` (line ~3897): when the row-sampling scan exceeds its
500ms budget, it computes the extrapolation base as
```ts
entryCount = isRocksDB
? primaryStore.getKeysCount({ start: undefined }) // <-- full-table native iteration
: primaryStore.getStats().entryCount;
```
The sampling scan itself yields per entry (`await rest()` → `setImmediate`) — that part is fine. The
problem is `getKeysCount`, which lands in rocksdb-js `Database::GetCount` / `TransactionHandle::getCount`
(`src/binding/database/database.cpp:791`, `src/binding/transaction/transaction_handle.cpp:509`):
```cpp
std::unique_ptr itHandle = std::make_unique(*dbHandle, itOptions);
while (itHandle->iterator->Valid()) {
++count;
itHandle->iterator->Next();
}
```
One synchronous napi call, `O(total keys incl. obsolete versions)`, on the JS thread.
## Production measurement (4-node cluster, harper-pro 5.1.26, Node v24.18.1)
A component-level periodic job calls `getRecordCount()` on two tables (816k and ~2.2M keys) every 15
minutes on worker index 0. Observed on a **freshly restarted, healthy** node:
1. **Harper's own throttle warning confirms the stall**: `JavaScript execution has taken too long and
is not allowing proper event queue cycling` (threshold: ≥3s event-loop delay) fires on that
worker's thread on the same 15-minute clock, for weeks, across worker generations (threadIds
1 → 17 → … → 145, all ≡ 1 mod 16 — and note those replaced generations are also why the warning
thread number keeps climbing).
2. **CPU profile of the worker across one run** (`Profiler` via inspector, 1ms sampling): longest
contiguous single-frame runs during the job window were **745ms** and **2,471ms** in the rocksdb-js
native frame, matching the two `getRecordCount` calls (816k and 2.2M keys). `getRecordCount @
Table.js:3663` visible in self-time.
3. **Live latency probe** (HEAD to the serve path, ~7/s, fresh connection each): baseline p50 6.1ms /
p99 8.3ms over 2,946 samples; during the job window, 53 of 54 samples stayed 5–8ms and the one
connection that landed on the blocked worker took **1,009ms** — arithmetically consistent with
arriving mid-way through the 2,471ms block.
On nodes affected by HarperFast/harper#2107 the same tables carry millions of retained obsolete versions, so this
iteration takes proportionally longer — our production cache-hit spike buckets (median 8.4s) on aged
nodes are this block, extended by dead-version skipping.
## Suggested fixes
- In `getRecordCount`, use the O(1) estimate for the extrapolation base on RocksDB:
`getEstimatedKeyCount()` (`rocksdb.estimate-num-keys`) instead of a full `getKeysCount` walk. The
result is already labelled an estimate (`estimatedRange`), so an estimated base is consistent with
the contract.
- In rocksdb-js, `getCount` deserves either an async variant that iterates off the JS thread, or
chunked iteration with periodic re-entry; a full-range synchronous count is a footgun for any
caller on a serving thread.
- Worth auditing other `getKeysCount`/`getCount` call sites with unbounded ranges.
Happy to provide the profiles/probe data.
Contributor guide
Research direction
Start in core/resources/Table.ts at getRecordCount(), then trace getKeysCount through rocksdb-js database.cpp and transaction_handle.cpp. Compare the existing sampling path with getEstimatedKeyCount and inspect other unbounded getKeysCount/getCount callers. Done means the count path no longer performs an unbroken full-table iteration on the serving thread while preserving the estimated-range behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, database, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100