HarperFast / HarperFast/harper

Post-failover write visibility divergence: surviving node's writes appear in indexed queries (without @createdTime) but not in collection scans

Open
#1,846 0 comments 0 reactions 0 assignees View on GitHub
area:replication area:storage bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

On a 2-node replicated cluster, in the window after one node dies (observed for 60+ seconds, ~1-in-5 reproduction under the cluster integration harness), the surviving node's own fresh writes intermittently enter a state where all of the following hold simultaneously:

- `put()` resolves successfully — 50+ consecutive writes at 1–5ms each, verified via trace logging of the writer;
- an **indexed-attribute REST query** (`GET /Table/?attr=value`) returns all of the new records — but with their `@createdTime` field **unset**;
- a **plain collection scan** (`GET /Table/`) remains frozen at pre-failure contents, across repeated polls and fresh requests, for the entire window.

Same node, same instant, two REST query plans, two different answers.

## Reproduction

Minimal fixture: a component with a 1s-interval job inserting unique-id records (`{ id: unique, node: server.hostname, firedAt: Date @createdTime }`) via the scheduler from #1828; boot 2 clustered nodes, deploy, kill the node currently running the writer, observe reads on the survivor. A polling read using the plain scan + `firedAt` filtering times out ~1 run in 5; the indexed `?node=` query sees the rows all along. The failover test being contributed to harper-pro alongside #1828 contains the dual-read-path diagnostics used to capture this (its assertion path has been switched to indexed-count polling so it gates only scheduler behavior).

Representative capture (survivor `127.0.0.4`, leader killed at `16:07:53.657Z`, snapshot 60s later):

```
plain scan (poll): rows=53 newest firedAt=16:07:53.519Z (pre-kill, frozen)
plain scan (fresh): rows=54 newest firedAt=16:07:53.519Z (still frozen)
indexed ?node=...: rows=50 (the post-kill writes) firedAt missing on all
writer trace log: ~55 puts completed in 1-5ms across the same window
```

SQL was deliberately excluded from the evidence chain.

## Hypothesis

Consistent with the documented "put's promise resolves before the commit callback runs" seam (resources/DESIGN.md): inserts reach the secondary index ahead of a commit/stamp stage that appears wedged on the dead peer during replication churn — `@createdTime` stamping and primary-scan visibility both ride the stage that stalls. Self-heals without intervention (subsequent runs usually pass).

Possibly related mechanisms worth checking during investigation: the committed-watermark pinning in HarperFast/rocksdb-js#668, the point-read/scan coherence class covered by #1760's single-node test, and the previously-fixed null-`@createdTime` in #329 (same symptom recurring via a different path). Context: #531 (documentation) covers within-transaction read-back semantics; this report is cross-request reads on a surviving node.

## Impact

Any application reading its own writes on a surviving node after a peer failure sees stale collection scans for the duration of the window — including the #1828 scheduler's own run-state bookkeeping (its engine tolerates this via in-memory anchors and idempotency, added defensively during review).

🤖 Filed by [Claude](https://claude.com/claude-code) on behalf of @jcohen-hdb; discovered while validating #1828's failover behavior.

Contributor guide

Open the contributing guide

Research direction

Start with resources/DESIGN.md and reproduce the dual-read behavior in the failover test described in the report. Compare the coherence coverage in #1760 and the @createdTime handling from #329, then verify that post-failover indexed queries and collection scans agree and that createdTime is present, with a regression test for the failure window.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.