HarperFast / HarperFast/harper
RocksDB subscription catch-up (startTime/previousCount) delivers patch/invalidate entries with no value — resumed consumers silently drop updates (5.2.x; fixed on main by #2409)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
On the 5.2.x line with the RocksDB transaction log, subscription **catch-up replay** (`subscribe({ startTime })` and the `previousCount` backfill) hands `patch` and `invalidate` audit entries to subscribers with **no usable value** (`value: undefined`), while the **live** path delivers the same write as a full-value event. Catch-up consumers therefore silently drop every partial-update write that happened while they were disconnected. Fixed on `main` (v5.3 line, unreleased) as a side effect of #2409; the whole released 5.2.x line (verified through the v5.2 branch tip, v5.2.7) is affected.
## Observed
Against a Fabric cluster running 5.2.3 (client-observable, reproduced in a Node client):
1. Subscriber holds a table-level subscription; disconnects.
2. An operations-API `update` runs against a row (writes a `patch` audit entry, `originatingOperation: update`). Studio edits take exactly this path.
3. Subscriber resumes with `startTime` = its last checkpoint.
4. The replayed event arrives as `{ type: 'patch', value: undefined, localTime: undefined }` — nothing to apply, so the consumer drops it. The same write observed live (connected at commit time) arrives as a full-value `put`.
REST `PUT` writes (full-record audit entries) replay fine, which makes the gap easy to misread as "some writes don't replicate": *offline edits made in Studio never arrive; REST edits do*.
## Root cause
The RocksDB transaction-log reader never stamps `localTime` onto decoded audit records — it only overwrites `version` with the log-key timestamp ([RocksTransactionLogStore.ts:416-417 @ v5.2](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/RocksTransactionLogStore.ts#L416-L417)).
Both collection catch-up paths pass that (undefined) `localTime` to `getValue` as the reconstruction time:
- `startTime` replay: [Table.ts:4210 @ v5.2](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/Table.ts#L4210)
- `previousCount` backfill: [Table.ts:4259 @ v5.2](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/Table.ts#L4259)
`getValue(store, fullRecord, auditTime)` returns the entry's own content only for full-record entries or `fullRecord: false` callers. For a partial-record entry (`patch`/`invalidate`, [auditStore.ts:370-372](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/auditStore.ts#L370-L372)) with `fullRecord: true` (every non-`rawEvents` subscriber, [Table.ts:4061](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/Table.ts#L4061)), it reconstructs via `getRecordAtTime(...)` **only when `auditTime` is truthy** ([auditStore.ts:666](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/auditStore.ts#L666)) — with `auditTime === undefined` it falls through and returns `undefined`.
The live listener is immune because it re-reads the just-committed entry from the primary store ([Table.ts:4134-4136](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/Table.ts#L4134-L4136)).
LMDB is immune because the audit-store iterator stamps `localTime` from the store key ([auditStore.ts:141](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/auditStore.ts#L141)). This is RocksDB-transaction-log only — i.e. the default Fabric configuration.
## Blast radius (5.2.x + RocksDB)
Broken — any consumer of table/collection catch-up crossing a `patch`/`invalidate` entry:
- **MQTT durable-session resume** (QoS > 0): `DurableSubscriptionsSession` resubscribes with the persisted `startTime` — resumed sessions silently lose every partial-update write from the offline window, violating the QoS ≥ 1 delivery expectation.
- **SSE / WS subscriptions passing `startTime`** (query params flow through `connect()` → `subscribe()`), i.e. reconnect catch-up.
- **Any component/user `subscribe({ startTime })`** — e.g. sync gateways doing checkpoint catch-up.
- **`previousCount` history backfills** (same `getValue` call).
Affected writes: everything that stores a partial audit entry — operations-API `update`, REST `PATCH`, SQL `UPDATE`; plus `invalidate` entries.
Not affected: live subscription events (re-read from primary store); single-record (`id`-level) replay, which walks the version chain with explicit per-entry timestamps ([Table.ts:4377](https://github.com/HarperFast/harper/blob/09fb3a75a/resources/Table.ts#L4377)); replication catch-up (raw/binary entries, applies patches as patches); `getHistory` (passes `auditRecord.version`, which on RocksDB *is* the log timestamp); LMDB-engine deployments.
## Fix — already on main
#2409 (b3235b8c6, merged 2026-08-31, milestone v5.3) stamps the decoded record in the reader:
```ts
auditRecord.version = timestamp;
auditRecord.localTime = timestamp; // <- this line is what un-breaks catch-up getValue
```
The PR's headline was a different symptom (source-fill events dropped by the live listener's staleness check, via the same missing-clock confusion); the `localTime` stamp incidentally fixes this replay bug too, and `main` verifiably delivers full values on catch-up.
## Suggested action
Cherry-pick b3235b8c6 to `v5.2` (backport PR milestoned `v5.2`). Dry-run merge shows the three source files (`RocksTransactionLogStore.ts`, `Table.ts`, `auditStore.ts`) apply cleanly; only `unitTests/resources/caching.test.js` conflicts (test-only). Worth adding while backporting: a replay assertion that a `patch` written by the operations API is delivered with a full value under the RocksDB engine — current `subscriptionReplay.test.js` asserts final values but not this wire shape specifically.
Contributor guide
Research direction
Start with RocksTransactionLogStore.ts around lines 416-417, then trace the startTime and previousCount paths in Table.ts at lines 4210 and 4259 into auditStore.ts. Run the RocksDB subscription replay coverage, especially subscriptionReplay.test.js, and add or update an assertion for an operations-API patch. Done means catch-up delivers a usable full value for patch and invalidate entries on the v5.2 line.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100