HarperFast / HarperFast/harper-pro
Sender-side subscription setup: the never-settling gate from #642 is bounded but not explained
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
[#642](https://github.com/HarperFast/harper-pro/issues/642) observed a replication subscription that stayed WebSocket-connected with fresh ping/pong for **more than two days** while the sender never got past its pre-`DB_SCHEMA` setup awaits. [#646](https://github.com/HarperFast/harper-pro/pull/646) bounds those awaits and recovers from the state — correct and needed on its own terms — but it does **not** explain why the promise failed to settle in the observed instance.
That matters because the two remedies are different. If the gate hung inside an async operation that can be fixed, the timeout is a recovery net over a live defect. If the gate was structurally unsettleable, the invariant to enforce is "a promise gating the replay loop must be settleable," and the timeout is the wrong layer to enforce it at.
## What is and isn't established
- **Established:** the failure shape (ping-alive, `DB_SCHEMA`/replay never start, zero received version, old durable cursor, one direction of one database, siblings healthy), and that bounding both gates recovers the link from its durable cursor.
- **Not established:** which of the two gates was stuck in the observed instance, and why. #646's regression coverage injects a never-settling authorization promise (`maybeStallSubscriptionSetupForTest`) — it proves the recovery net works, not that the natural cause is understood or gone.
## The two gates, and what "never settles" means for each
`SUBSCRIPTION_REQUEST` handling in `replication/replicationConnection.ts` awaits, in order:
1. **Dynamic send-authorization** — `getHDBNodeTable().subscribe(authorization.name)`. A fresh promise per connection.
2. **Database subscription** — `dbSubscriptions.get(databaseName)`, or a placeholder created by `createPendingDatabaseSubscription()` and registered in that map. It resolves only when `Replicator.subscribe()` runs for the first table of that database **on this thread**.
Gate 2 is the stronger candidate, because a never-settling outcome there requires no hang at all — only that `Replicator.subscribe()` never runs for that database on the accepting worker. The placeholder is then a promise that is *structurally* unsettleable, globally retained in `dbSubscriptions` until the database is removed. The observed selectivity (one direction, one database, siblings fine) fits a per-(database, thread) registration gap better than it fits a generic async hang.
Candidates for gate 1, if the log points there instead:
- Theme H of #430 — a sync consumer of a `MaybePromise` storage read on the `hdb_nodes` path.
- Eventually-consistent role/schema propagation across workers ([harper#1497](https://github.com/HarperFast/harper/issues/1497)) leaving the subscribe call waiting on state that arrives on another worker.
## Why this is now cheap to answer
#646 logs **which gate timed out** and classifies timeout vs. resolved-without-a-subscription. So the first occurrence after #646 ships names the gate directly, in the field or in the nightly cluster suite. Until then the question is open, not unanswerable.
## Acceptance criteria
- The stuck gate in the observed instance (or the first post-#646 occurrence) is named from the classification log.
- The reason it did not settle is explained at the level of the code path, not the symptom.
- A decision is recorded: either a structural fix (settleable/evictable placeholder, fast-fail for a database this node does not host, or a fix to whatever hung in `hdb_nodes` subscribe) — or an explicit "bounded-and-retry is the correct terminal state for this cause," with the reasoning.
- If the cause is the gate-2 registration gap, coverage exists for it end-to-end (the current integration test exercises the authorization gate only).
Follow-up to #642 · #646 · part of #430.
🤖 Filed by Claude (Opus 5) on behalf of Kris.
Contributor guide
Research direction
Start in replication/replicationConnection.ts and inspect the SUBSCRIPTION_REQUEST gates in order. Review #646's timeout classification and the current integration test that injects maybeStallSubscriptionSetupForTest, then use the first classified occurrence to identify the gate and code-path cause. Done means explaining why it failed to settle and recording the structural fix or why bounded retry is correct, with gate-two end-to-end coverage if applicable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100