HarperFast / HarperFast/harper

@expiresAt per-record TTL never evicts on RocksDB (default engine) — index.getValues() is LMDB-only

Open
#1,481 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A table with an `@expiresAt` field (per-record absolute-expiry timestamp — the session/token/cache pattern) **never evicts expired records on RocksDB** (the default engine). The per-record eviction sweep throws on every pass and is silently swallowed, so records past their `expiresAt` accumulate forever **and** remain live + readable indefinitely. Works correctly on LMDB.

## Mechanism

`resources/Table.ts:5293` `runRecordExpirationEviction()` calls `index.getValues(key)` — an LMDB-specific dupsort API that does not exist on RocksDB index stores. On RocksDB every ~60s sweep throws `TypeError: index.getValues is not a function`; the error is caught + logged but otherwise swallowed, so the sweep makes no progress.

There is no lazy-on-read fallback for an `@expiresAt`-only table: the entry-metadata `expiresAt` is `-1` (no `@table(expiration:)`), and `resources/RecordEncoder.ts:708-709` only sets `HAS_EXPIRATION` when `expiresAt >= 0`. So on RocksDB the records are neither swept nor lazily expired.

## Impact

- **Storage leak:** `@expiresAt` records grow unbounded on RocksDB.
- **Correctness / security-adjacent:** a "session"/"token" table using `@expiresAt` keeps serving records that should have expired — an expired token never expires.
- Affects the **default engine**, so it hits most deployments.

## Fix

Replace the LMDB-only `index.getValues(key)` with a RocksDB-compatible `index.getRange(...)` iteration that reads `entry.value` as the primary key (the RocksDB index model), so the per-record sweep works on both engines.

## Secondary footgun (both engines)

A full-record `PUT` that omits the `expiresAt` field **clears** it (the record then never expires). A `PATCH` of a non-expiry field correctly preserves it. (And `@table(expiration:N)` + `@expiresAt` together → the table-level TTL wins.) Worth documenting that a session-renewal-via-PUT must re-include `expiresAt`.

## Reproduction

```
cd /home/kzyp/dev/harper # main @ 28db4fde4 (v5.1.12)
npm run build
npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts" # RocksDB → broken (records never expire)
HARPER_STORAGE_ENGINE=lmdb npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts" # LMDB → works
```

Found by the qa-explorer exploratory campaign. — filed by KrAIs (Claude) on Kris's behalf

Contributor guide

Open the contributing guide

Research direction

Start in resources/Table.ts:5293 and inspect the expiration sweep's index access, then review resources/RecordEncoder.ts:708-709 for expiration metadata behavior. Run integrationTests/qa-scratch/qa365-expiresat.test.ts with the default RocksDB engine and with LMDB; done means expired @expiresAt records are evicted on both engines and the PUT behavior is covered or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.