HarperFast / HarperFast/harper
Post-failover write visibility divergence: surviving node's writes appear in indexed queries (without @createdTime) but not in collection scans
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
On a 2-node replicated cluster, in the window after one node dies (observed for 60+ seconds, ~1-in-5 reproduction under the cluster integration harness), the surviving node's own fresh writes intermittently enter a state where all of the following hold simultaneously:
- `put()` resolves successfully — 50+ consecutive writes at 1–5ms each, verified via trace logging of the writer;
- an **indexed-attribute REST query** (`GET /Table/?attr=value`) returns all of the new records — but with their `@createdTime` field **unset**;
- a **plain collection scan** (`GET /Table/`) remains frozen at pre-failure contents, across repeated polls and fresh requests, for the entire window.
Same node, same instant, two REST query plans, two different answers.
## Reproduction
Minimal fixture: a component with a 1s-interval job inserting unique-id records (`{ id: unique, node: server.hostname, firedAt: Date @createdTime }`) via the scheduler from #1828; boot 2 clustered nodes, deploy, kill the node currently running the writer, observe reads on the survivor. A polling read using the plain scan + `firedAt` filtering times out ~1 run in 5; the indexed `?node=` query sees the rows all along. The failover test being contributed to harper-pro alongside #1828 contains the dual-read-path diagnostics used to capture this (its assertion path has been switched to indexed-count polling so it gates only scheduler behavior).
Representative capture (survivor `127.0.0.4`, leader killed at `16:07:53.657Z`, snapshot 60s later):
```
plain scan (poll): rows=53 newest firedAt=16:07:53.519Z (pre-kill, frozen)
plain scan (fresh): rows=54 newest firedAt=16:07:53.519Z (still frozen)
indexed ?node=...: rows=50 (the post-kill writes) firedAt missing on all
writer trace log: ~55 puts completed in 1-5ms across the same window
```
SQL was deliberately excluded from the evidence chain.
## Hypothesis
Consistent with the documented "put's promise resolves before the commit callback runs" seam (resources/DESIGN.md): inserts reach the secondary index ahead of a commit/stamp stage that appears wedged on the dead peer during replication churn — `@createdTime` stamping and primary-scan visibility both ride the stage that stalls. Self-heals without intervention (subsequent runs usually pass).
Possibly related mechanisms worth checking during investigation: the committed-watermark pinning in HarperFast/rocksdb-js#668, the point-read/scan coherence class covered by #1760's single-node test, and the previously-fixed null-`@createdTime` in #329 (same symptom recurring via a different path). Context: #531 (documentation) covers within-transaction read-back semantics; this report is cross-request reads on a surviving node.
## Impact
Any application reading its own writes on a surviving node after a peer failure sees stale collection scans for the duration of the window — including the #1828 scheduler's own run-state bookkeeping (its engine tolerates this via in-memory anchors and idempotency, added defensively during review).
🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @jcohen-hdb; discovered while validating #1828's failover behavior.
Contributor guide
Research direction
Start with resources/DESIGN.md and reproduce the dual-read behavior in the failover test described in the report. Compare the coherence coverage in #1760 and the @createdTime handling from #329, then verify that post-failover indexed queries and collection scans agree and that createdTime is present, with a regression test for the failure window.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100