coordinator: bootstrap waits for a deleted capture until captureRemoveTTL expires

Coordinator bootstrap can remain blocked on a capture whose etcd key has already been deleted, until captureRemoveTTL expires. All surviving captures may have returned their bootstrap responses during this wait.

Observed with the new architecture at commit 52b8f42a5db36ea0ab0c5e60aefed13a4d8a49c5 during a two-capture rolling restart.

Observed behavior

The following relative timeline summarizes the observed sequence. t=0 is when the new coordinator observes deletion of the old capture's etcd key:

There is approximately four seconds of additional bootstrap waiting after the replacement's response arrives. This is the remaining portion of the capture-removal delay. The overall checkpoint-lag spike also includes other restart and recovery work; it cannot all be attributed to this wait.

Mechanism

At this revision:

• GlobalReactorState retains a deleted capture in Captures for max(captureSessionTTL / 2, 10s) before publishing its removal.
• Controller.onNodeChanged supplies the NodeManager membership view to the bootstrapper.
• Bootstrapper.collectBootstrapResponses requires a response from every tracked node. A replacement with a new capture ID cannot satisfy the old capture's pending response.

As a result, the scheduling safety delay also blocks coordinator initialization.

Proposed minimal reproduction

These reduced reproductions have not yet been run. The observation above comes from the rolling-restart execution.

Deterministic component-level test

1. Initialize membership and a coordinator bootstrapper with captures A and B.
2. Let B return its bootstrap response; leave A's response pending.
3. Deliver A's capture-key deletion event to GlobalReactorState and process pending changes before captureRemoveTTL expires.
4. Assert that A remains in the membership view and bootstrap remains incomplete despite B having responded.
5. Advance the pending-removal age past captureRemoveTTL, process membership changes, and pass the resulting view to the bootstrapper.
6. Assert that removing A allows bootstrap to complete without any response from A.

This isolates the wait without Kafka or a workload.

End-to-end validation

Use two TiCDC captures, one changefeed, and one table with continuous writes. During coordinator handover, ensure the old capture is included in the new coordinator's bootstrap set and exits before replying. Use a test synchronization point to control this ordering; an ordinary restart may not reproduce it on every attempt.

Measure capture-key deletion observation, receipt of all surviving-node responses, publication of old-node removal, and coordinator bootstrap completion. Record checkpoint lag to assess the replication impact.

Expected behavior / discussion

Evaluate whether coordinator bootstrap can collect and restore surviving-node state without waiting for a response from a known-deleted capture, while separately enforcing the safety delay before its tasks can be taken over.

The existing capture-write-lease design uses captureRemoveTTL to separate old-writer and replacement-writer admission windows. Any optimization must preserve that guarantee and the checkpoint/task-ownership constraints. Completing bootstrap earlier requires a safety assessment; this issue does not establish that removing or bypassing the delay is safe.