HarperFast / HarperFast/harper
@expiresAt per-record TTL never evicts on RocksDB (default engine) — index.getValues() is LMDB-only
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A table with an `@expiresAt` field (per-record absolute-expiry timestamp — the session/token/cache pattern) **never evicts expired records on RocksDB** (the default engine). The per-record eviction sweep throws on every pass and is silently swallowed, so records past their `expiresAt` accumulate forever **and** remain live + readable indefinitely. Works correctly on LMDB.
## Mechanism
`resources/Table.ts:5293` `runRecordExpirationEviction()` calls `index.getValues(key)` — an LMDB-specific dupsort API that does not exist on RocksDB index stores. On RocksDB every ~60s sweep throws `TypeError: index.getValues is not a function`; the error is caught + logged but otherwise swallowed, so the sweep makes no progress.
There is no lazy-on-read fallback for an `@expiresAt`-only table: the entry-metadata `expiresAt` is `-1` (no `@table(expiration:)`), and `resources/RecordEncoder.ts:708-709` only sets `HAS_EXPIRATION` when `expiresAt >= 0`. So on RocksDB the records are neither swept nor lazily expired.
## Impact
- **Storage leak:** `@expiresAt` records grow unbounded on RocksDB.
- **Correctness / security-adjacent:** a "session"/"token" table using `@expiresAt` keeps serving records that should have expired — an expired token never expires.
- Affects the **default engine**, so it hits most deployments.
## Fix
Replace the LMDB-only `index.getValues(key)` with a RocksDB-compatible `index.getRange(...)` iteration that reads `entry.value` as the primary key (the RocksDB index model), so the per-record sweep works on both engines.
## Secondary footgun (both engines)
A full-record `PUT` that omits the `expiresAt` field **clears** it (the record then never expires). A `PATCH` of a non-expiry field correctly preserves it. (And `@table(expiration:N)` + `@expiresAt` together → the table-level TTL wins.) Worth documenting that a session-renewal-via-PUT must re-include `expiresAt`.
## Reproduction
```
cd /home/kzyp/dev/harper # main @ 28db4fde4 (v5.1.12)
npm run build
npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts" # RocksDB → broken (records never expire)
HARPER_STORAGE_ENGINE=lmdb npm run test:integration -- "integrationTests/qa-scratch/qa365-expiresat.test.ts" # LMDB → works
```
Found by the qa-explorer exploratory campaign. — filed by KrAIs (Claude) on Kris's behalf
Contributor guide
Research direction
Start in resources/Table.ts:5293 and inspect the expiration sweep's index access, then review resources/RecordEncoder.ts:708-709 for expiration metadata behavior. Run integrationTests/qa-scratch/qa365-expiresat.test.ts with the default RocksDB engine and with LMDB; done means expired @expiresAt records are evicted on both engines and the PUT behavior is covered or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100