HarperFast / HarperFast/prerender-plugin
Orphaned nextRenderTime index entries make the claim scan non-monotonic; one break at row 0 silently stalls a node's entire backlog (2 of 4 nodes affected)
- Dominant language
- JavaScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 9h 39m
- Merged PRs (30d)
- 39
Description
## Summary
A claim pass that sees **no due rows** advances the floor to `nowMinute - guard`. That is correct on a
caught-up queue, but it is an unconditional forward jump with no way back: any row that later appears
below the floor is invisible to every subsequent pass, because the scan is a single
`nextRenderTime >= floor` condition. Nothing in the claim path can lower the floor again.
What makes the pass see no due rows is a single **orphaned secondary-index entry** pointing at a row
whose stored value has since moved forward. The walk visits the orphan, loads the record, and yields a
**future** value at the head of an otherwise ascending scan. The pass breaks on it at iteration 0 —
every pass, forever — and then ratchets the floor past everything below.
Observed in production as one node of four silently ceasing to render a growing slice of its shard:
**92,873 rows stranded below the floor across 2,101 distinct due-minutes**, oldest due 2026-08-01 — 18
days. **Three orphaned entries were withholding ~93k rows of work.**
Two defects:
1. **Index maintenance leaves an entry that no longer describes the record.** Upstream — see status below.
2. **The claim path trusts the index ordering absolutely.** One `break` on the first future row turns a
single bad entry into a permanent, silent, whole-node queue stall.
**Defect 2 is the one worth fixing.** It is cheap, local to this plugin, and it makes the claim path
robust to any ordering violation regardless of who caused it. It is also still entirely present — see
"Where defect 2 lives today".
## Upstream status — updated 2026-09-16
`HarperFast/harper#2211`, filed alongside this issue, is **closed**: fixed 2026-08-20 in `efd5e535`,
*"keep same-key writes in staging order so a delete can't strip a live record's indexes"*. A replicated
`delete K; put K` transaction could leave the follower holding a live, correct record with **none** of
its secondary index entries.
**That is not the signature measured here, and it is not established that the fix covers this one.**
The state on this cluster was an entry at the record's **old** value *and no entry at the new one* — a
walk starting at the record's own `nextRenderTime` did not return the key, while a point read returned
the later value. That matches the `updateIndices` early-`continue`:
```js
if (value === existingValue && !isIndexing) continue; // no index work at all, in either direction
```
— the index untouched while the record moved forward — rather than a delete stripping entries on the
way through. So: re-verify against a Harper build carrying `efd5e535` before treating defect 1 as gone.
Defect 2 is unaffected either way, and is what this issue now tracks.
## Root cause, as measured
Scanning each node's `nextRenderTime` index from 0, `limit: 200`, ascending:
| node | monotonic violations | head of walk | future rows in window | overdue in window |
|---|---|---|---|---|
| **cd5** | 1 | 2026-08-20T11:24 | 2 | **198 / 200** |
| **yc0** | 1 | 2026-08-29T23:03 | 1 | **199 / 200** |
| e9v | 0 | 2026-08-19T16:01 | 158 | 42 |
| v3t | 0 | 2026-08-19T16:01 | 126 | 74 |
**Two of four nodes affected, independently.** The orphans, with the position bisected out of the index:
```
cd5 |mobile record 2026-08-20T11:24 indexed 2026-08-14T02:06 fromSitemap=true
cd5 |desktop record 2026-08-20T11:24 indexed 2026-08-14T02:06 fromSitemap=true
yc0 |desktop record 2026-08-29T23:03 indexed 2026-08-15T23:02 fromSitemap=false
```
(Those keys are pre-`v0.66.0` shapes — `RenderSchedule` is keyed by URL now, one row per URL rather
than one per URL+device. The mechanism is unchanged: residency, the index, and the floor all work the
same way on the new key.)
Frequency was roughly 3 rows in ~1.3M over several days. The precise trigger was never established —
the affected rows have since been repaired and re-rendered, and dating the divergent writes from
`record.nextRenderTime - interval` is unreliable because the demand ladder rewrites intervals.
### The aborted transactions were not the cause
Recorded so nobody re-walks it. Record and index writes share one transaction
(`updateIndices(id, existingRecord, recordToStore, transaction && { transaction })`, with that same
transaction passed into `index.remove(...)`/add), so an abort rolls back both and cannot half-apply.
And the aborts named only `PrerenderedPage/` and `VisitFilter/` while the orphans were in
`render_schedule` — different databases, independent commits.
## Where defect 2 lives today
Still unguarded, and now in **two** places rather than one — v0.50.0's ready-set sweep took over
observing the index on nodes serving claims from the ready set, and it repeats the same shape:
- `util/renderSchedule.js` — the claim pass breaks on the first future row, then
`floorTo = Math.max(0, observed ?? nowMinute - guard)`.
- `util/renderSchedule.js` — the ready sweep breaks the same way, then
`advanceFloor(floorFrom, Math.max(0, firstDueMinute ?? nowMinute - guardMinutes()))`.
So the ready set did not remove this: a node whose index carries one orphan at the head strands its
backlog through whichever path is serving claims.
### Why every recovery fails
- **`reset-claim-floor` is futile.** Confirmed live: reset, and the floor was back at `now - guard`
immediately. From a zeroed floor the walk returns the same orphan first and the pass breaks at
iteration 0 again.
- **`claimFloor.resetInterval`** is the same reset, so equally futile.
- **`unpinAfter`** is irrelevant — the pass never gets far enough to name a pinned row.
- **`claimFloor.enabled: false` is worse than futile.** Over 75s it recovered **zero** stranded rows and
the overdue count accelerated from ~7/min drift to ~200/min, because `maybeUnpinFloor` early-returns
when the floor is disabled (`if (!config.queue.claimFloor.enabled) return null;` — still there today)
and `runClaimPass` then calls `resetFloor()` every pass, restarting from the absolute index minimum
while still capped at ~100 rows.
Current-minute work still renders (a pass whose window happens to open on a due row reports
`sawDue: true`), which is why the node looks healthy and `QueueStatus` reads `queued`.
## Ruled out
- **Tie-pileup at one minute** — the stranded set spans 2,101 distinct minutes at ~44 rows/minute.
- **A single poison row wedging the head** — the head of the index is ordinary product URLs.
- **`unpinAfter` not firing** — pin age lives in the SharedBuffer and resets on restart; node uptime was
under the 1h default, so zero unpin warnings is expected.
## Asks — restated against the tree as of 2026-09-16
1. **Bound the forward advance. OPEN.** A pass should only advance the floor as far as it actually
scanned — to `earliestNotYetDueMinute` when it reached a not-yet-due row, and not at all when the
scan was cap-truncated. `observed ?? nowMinute - guard` is the defect, and it is unchanged in both
the claim pass and the ready sweep. Note the *signal* now exists where it did not before: the pass
computes `scanTruncated` (`rows.length >= scanLimit && earliestNotYetDueMinute === 0`) and reports
it. It just does not gate the advance on it.
2. **`advanceFloor`'s contract. LANDED.** It is now a CAS against the value the pass started from,
abandoning on conflict — *"a conflict means a funnel write lowered the floor for a row this pass
never saw"*. That is the assertion this ask wanted; it does not by itself stop the empty-scan
ratchet, which is ask 1.
3. **Document or repair the kill switch. OPEN.** `claimFloor.enabled: false` still disables
`maybeUnpinFloor` (`renderSchedule.js`, the second guard in that function). Either stop it doing
that, or say plainly in the option text that it does — it is presented as a safe "changes nothing
else" switch and is a downgrade for a node already stranded.
4. **Make `below_floor` honest. OPEN.** It is still counted inside a `management.scanCap`-bounded walk,
so it saturates at the cap: a node with 92,873 stranded rows and a node with 1,998 publish the same
number. `backlogSnapshot` already computes `truncated` for the walk — emit it alongside, or count
`belowFloor` uncapped, so an alarm can tell "1,998" from "at least 2,000 and we stopped counting".
5. **Per-node `ConfigOverride` scoping. OPEN.** `ConfigOverride.path` is still the sole primary key and
the table is replicated and not residency-pinned, so every lever for a node-local condition is
cluster-wide. `QueueControl` already implements the pattern — `scope` as primary key, `'all'` plus
per-hostname rows, delete to inherit.
One unexplained detail still worth a look: `oldestBelowFloorMs` read 2026-08-14T02:07 while the table's
true minimum `nextRenderTime` was 2026-08-01T00:57. If the walk really is ascending from the absolute
minimum, those should agree.
## Production evidence
Plugin 0.49.0, harper-pro 5.2.3, 4-node cluster, ~1.6M keys, all `queue.*` at defaults.
```
rows below floor 92,873 <- invisible to claim
rows floor..now 2 <- all claim can see
distinct due-minutes 2,101
oldest stranded row 2026-08-01T00:57Z
```
Other three nodes: 507 / 3 / 1 overdue. This bites whichever node's floor gets ahead, not the cluster
uniformly. Claim was granting nothing — `granted 0 of 5` in 256 of 270 warned passes.
#111 is the companion: `writeSchedule` lowers only the *writing* node's floor while the row is
residency-routed to its **owner**, so ~75% of writes leave the owner's floor untouched — which is what
keeps depositing rows underneath a floor that has already moved past them.
Contributor guide
Research direction
Start in util/renderSchedule.js with the claim pass and ready sweep, tracing how scanTruncated, earliestNotYetDueMinute, and observed determine floor advancement. Also inspect the claimFloor.enabled guard and backlogSnapshot's truncated value. Done means the open behaviors are bounded and reported honestly without allowing an orphaned index entry to strand the backlog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100