Cross-node clock skew silently consumes the system event overlap window

Problem

System event delivery between cluster nodes depends on the nodes' clocks agreeing, and nothing detects it when they don't.

created is stamped by the authoring node (SystemEvent constructor: creationDate == null ? new Date() : creationDate) and written to system_event as-is. The reading node derives its read floor from its own clock (SystemEventsCursorTracker.beginPoll, readFloor = cursor - overlapWindow). The query is created >= readFloor, so any skew between the two clocks is subtracted directly from the overlap window:

effective commit-lag tolerance = SYSTEM_EVENTS_OVERLAP_WINDOW_SECONDS - peer clock skew

At the 120s default, a node whose clock runs 3 minutes behind its peers publishes events that those peers never see — every event, permanently, with no warning.

Why this is hard to spot

The in-product reconciliation cannot detect it. It compares a node against itself (WHERE server_id = <me>), and a skewed node observes its own events normally. So every node reports 0% loss while cross-node delivery is entirely broken.

There is no other signal. The commit-lag warning does not fire either, because from the reading node's perspective the events simply never appear.

Current state

Found during review of #37288 (fix for #36827). That PR documents the limitation but deliberately does not change behaviour:

• SystemEventsCursorTracker class javadoc states the dependency (it previously claimed the opposite).
• docs/backend/SYSTEM_EVENTS.md §1 "What is not" states the budget, the NTP requirement, and the diagnostic note.
• Two unit tests in SystemEventsCursorTest pin the budget so it is checkable.

Adding tolerance or detection was left out of that PR because it is a behaviour change resting on a design question, not a documentation fix.

The design question

Whose clock is authoritative? The options are not equivalent:

1. Detect and warn only. Compare each node's clock against a shared reference (e.g. the database clock, or the mod_date of peer cursor rows in system_event_cursor) and warn past a threshold. Cheapest, changes no delivery behaviour, turns a silent failure into a loud one. Does not prevent loss.
2. Stamp created from the database rather than the publishing JVM. Removes the dependency at the source, since one clock then orders everything. Touches the write path and the meaning of created for existing rows, so it needs care around the migration and around any consumer reading that column.
3. Widen the window to cover expected skew. Not really a fix — it trades memory and re-delivery for a budget that is still finite and still silent when exceeded.

Option 1 is the smallest useful step and composes with either of the others. Option 2 is the actual fix.

Suggested acceptance criteria

• A node whose clock differs from the cluster reference by more than a configurable threshold produces a WARN naming the measured offset.
• The condition is visible without reading logs on every node (reconciliation output is one candidate home).
• A test that pins detection at the threshold boundary.

Notes

• Related: #36827 (the delivery fix), #37288 (the PR where this was found).
• SYSTEM_EVENTS_OVERLAP_WINDOW_SECONDS currently doubles as both the commit-lag budget and the skew budget; whichever option is chosen, it would help to stop conflating the two.

Start with SystemEventsCursorTracker and the two SystemEventsCursorTest cases, then read docs/backend/SYSTEM_EVENTS.md §1 and the related issues #36827 and #37288. Resolve which clock should be authoritative before choosing detection or delivery changes. Done means a configurable-threshold WARN names the measured offset, the condition is visible outside per-node logs, and a boundary test passes.