Replace ReindexThread's 250ms pause poll with wait/signal, including a cluster-level unpause wake

Summary

ReindexThread's pause loop polls every 250 ms (REINDEX_THREAD_SLEEP) to discover that it has been unpaused. Replace that poll with a proper wait/signal, including a cluster-level wake — because the poll is currently the only mechanism by which one node learns that another node queued work.

Current behaviour

private void sleep() {
    while (state.get() == ThreadState.PAUSED) {
        if (shutdownRequested()) { return; }
        if (!waitFor(SLEEP)) { requestStop(); return; }   // 250 ms
        Long restartTime = (Long) cache.get().get(REINDEX_THREAD_PAUSED);
        if (restartTime == null || restartTime < System.currentTimeMillis()) {
            state.compareAndSet(ThreadState.PAUSED, ThreadState.RUNNING);
        }
    }
}

A paused worker wakes 4 times per second (~345,600/day), each time doing a SystemCache lookup, purely to notice a state change that something else already knows about. Every wakeup is also a timer interrupt working against the CPU's idle states.

Note the cost is the polling, not the sleeping: a sleeping platform thread uses no CPU and only its stack. This is not an urgent performance problem — it is avoidable work on an idle node.

The cluster half — why this is not just a local change

pause() and unpause() are asymmetric today:

• pause() → SystemCache.put(...) — node-local
• unpauseImpl() → SystemCache.remove(...) → ChainableCacheAdministratorImpl.remove — broadcasts a cluster-wide invalidation

So when node A unpauses, node B's marker is cleared too — but node B only notices on its next 250 ms poll. The poll is doing double duty: it is both the local wait and the cross-node wake.

Removing the poll without adding a cluster signal would regress multi-node behaviour: node B would sit paused until its pause deadline expired, delaying indexing of anything it has claimed. Any implementation must cover both halves.

Proposed approach

1. Local: replace the poll with Condition.await(timeout) / LockSupport.parkNanos, signalled directly by unpauseImpl(). Keeps the pause deadline as the timeout so the existing REINDEX_THREAD_PAUSE_IN_MINUTES semantics are preserved.
2. Cluster: publish a wake event so other nodes are pushed rather than polling. DotPubSubTopic already exists with three implementors to follow — CacheTransportTopic, OsgiRestartTopic, ClusterManagementTopic — so a small ReindexWakeTopic fits the established pattern. Alternatively, hook the existing REINDEX_THREAD_PAUSED invalidation already crossing the transport and signal the local worker on receipt.

Expected outcome: zero wakeups while paused instead of 4/s, and unpause latency drops from up to 250 ms to effectively immediate, on every node.

Why now

#36922 (PR #37295) made this tractable. All three waits in ReindexThread now route through a single interrupt-aware waitFor(long) that restores the interrupt status. Previously they used ThreadUtils.sleep, which swallows both InterruptedException and the interrupt flag — a parked worker could not be woken at all, so no signalling scheme would have been reliable.

Related

• A cluster wake would also partially address the node-local recovery limitation documented in #36922: today unpauseImpl() consults only the local node's liveness, so node A's unpause cannot restart node B's dead worker. If node B were signalled, it could check its own liveness on receipt. (Full cross-node liveness detection is larger and still out of scope.)
• Virtual threads were considered and are not the answer here. They win with many threads blocked on socket I/O; this is one worker per JVM, so unmounting saves a single platform thread. Per docs/backend/VIRTUAL_THREADS.md, this worker's mixed JDBC + file/cache profile is also a poor candidate — see the H22 migration reverted in #36900 after #36892.

Acceptance Criteria

• A paused ReindexThread performs no periodic wakeups; it waits until signalled or until the pause deadline expires
• unpauseImpl() signals the local worker directly rather than relying on it to observe a cache change
• An unpause on one node wakes paused workers on the other nodes without polling
• Existing semantics preserved: REINDEX_THREAD_PAUSE_IN_MINUTES deadline, REINDEX_THREAD_SLEEP (or its documented removal), and shutdown/interrupt behaviour from #36922
• Multi-node test showing content queued on node A is indexed promptly with no 250 ms poll involved

Start in ReindexThread and trace its interrupt-aware waitFor(long), pause(), and unpauseImpl() paths, then review the DotPubSubTopic implementations CacheTransportTopic, OsgiRestartTopic, and ClusterManagementTopic. Check the behavior described in #36922 and PR #37295 before choosing the local and cluster wake mechanism. Done means no periodic paused polling, preserved deadline and shutdown behavior, and a multi-node test proving prompt wakeup.