HarperFast / HarperFast/prerender-plugin

writeSchedule lowers only the writing node's claim floor, but rows are residency-routed to their owner — ~75% of writes can strand rows silently

Đang mở
#111 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
JavaScript
Star
0
Fork
0
Merge trung bình
9 giờ 39 phút
Pull request đã merge (30 ngày)
39

Mô tả

> **Rewritten 2026-08-19.** The first version of this issue blamed *replication*. That was wrong —
> `RenderSchedule` is not replicated; rows are residency-routed and each key lives on exactly one
> node (verified: a 27-key probe returned 1/16/10/0 across four nodes, perfectly disjoint). The
> actual mechanism is below, and it is worse, because it fires from ordinary in-plugin writes.

## Summary

`writeSchedule` lowers the claim floor of the node **executing the write**. But the row it just
wrote is residency-routed to whichever node **owns** that cacheKey. On a 4-node cluster ~75% of
writes therefore lower a floor belonging to the wrong node, and the owner's floor is never lowered
at all.

Any row filed with a due minute older than `queue.claimFloor.guard` (default 5 min) lands **below
its owner's floor** and is never claimed again — silently, permanently, from a fully funnel-routed
write.

## The mechanism

```js
export const writeSchedule = async (cacheKey, { nextRenderTime, fromSitemap } = {}) => {
...
await scheduleTable().put(cacheKey, { nextRenderTime, fromSitemap });
lowerFloorFor(nextRenderTime); // ← local SharedBuffer, i.e. THIS node's floor
};
```

`lowerFloorFor` → `leaseTable().lowerFloorTo(...)` is an `Atomics` CAS-min against the local
`coordination.SharedBuffer` (`replicate: false`, node-local by construction). The `put`, however,
is residency-routed — and per the module comment on `writeSchedule`, deliberately does not block
on the owner:

> a write to a residency-pinned key this node does not own does NOT block on the owner (measured:
> 500 writes in 10.7 ms, mean 0.021 ms, against residency pinned to a node that does not exist)

Ownership is `getResidencyByUrl` → `fnv1a32(\`${url}|${node}\`)` rendezvous hashing, so for any
given writer 3 of 4 keys are owned elsewhere.

## This is already documented in the tree

`resources/Target.js`, in `revalidate`'s phase 2 — the hazard is known and currently mitigated
only by writers remembering to recompute the minute per URL:

> THE CURRENT MINUTE, PER URL — never captured once for the whole sweep. Phase 2 writes up to
> `scan.collectCap` × devices rows with a `PrerenderedPage.get` per key, which at scale takes tens
> of minutes. Rows are residency-routed, so ~75% land on nodes whose claim floor this process
> cannot lower and which hold it at `nowMinute − queue.claimFloor.guard`: every row filed with a
> minute more than the guard band old lands BELOW the owner's floor and is never claimed again —
> silently, from a fully funnel-routed in-plugin write, and permanently where `resetInterval: 0`.
> `Sitemap.js` already computes it per entry for the same reason.

So the invariant "never let more than `guard` elapse between computing a due minute and writing
it" is load-bearing across every schedule writer, enforced nowhere, and unobservable when broken.
Two call sites currently carry hand-written comments to remember it.

## Impact

Observed on a 4-node production cluster (plugin 0.49.0 / harper-pro 5.2.3). One node accumulated
**92,873 rows below its floor across 2,101 distinct due-minutes**, oldest due 2026-08-01 — 18 days
of a growing slice of its shard silently not rendering, while the node reported healthy and kept
serving. Peer nodes were at 507 / 3 / 1 overdue.

Downstream, those keys held damaged blobs that could only be replaced by a re-render, so
replication base copies from the affected node latched indefinitely (harper-pro#699 territory) —
a queue bug surfacing as a storage symptom.

## Why the nominated recovery does not cover it

`queue.claimFloor.resetInterval`'s docstring names the out-of-plugin write paths it exists to
cover — the operations API and the exported REST surface. Residency-routed **in-plugin** writes
are not mentioned, and they are the high-volume path. The reset is also worker-0-gated inside
`syncQueueState` (see the open question in #110).

## Possible directions

1. **Lower the owner's floor, not the writer's.** The write already routes to the owner; the floor
lowering needs to ride with it rather than being applied locally. That is the fix that makes
the invariant unnecessary.
2. **Bound the forward advance** so a floor can never outrun rows the pass never scanned (#110).
That alone would stop the stranding even if the lowering stays local.
3. **Make the below-floor condition observable.** The backlog snapshot already computes
`below_floor`; a node with essentially its whole due backlog below the floor should alarm.
4. Failing 1, at minimum assert the invariant at the funnel — reject or warn on a `writeSchedule`
whose `nextRenderTime` minute is already older than `guard`, so a slow batch fails loudly
instead of silently stranding rows on a peer.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start at writeSchedule and trace lowerFloorFor through leaseTable().lowerFloorTo(...) and the local coordination.SharedBuffer. Then read resources/Target.js, Sitemap.js, and getResidencyByUrl to compare existing per-URL handling with residency ownership. Done should include an agreed direction, coverage for ordinary in-plugin writes, and evidence that rows are not left below the owning node's floor.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
javascript
Lĩnh vực
backend, distributed-systems
Loại issue
Lỗi
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.