release-nextgen-202603: HA (PD leader partition) still reproducibly freezes changefeed checkpoint after capture replacement

Bug Report

Summary

In the TiCDC new architecture, a PD leader ↔ PD followers network partition (HA fault injection) makes a capture exit abnormally (capture suicide + nil-pointer panic during server close). After the new capture takes over, the changefeed checkpoint is permanently frozen: checkpointTs / resolvedTs stop advancing, and the checkpoint lag grows linearly (~3.3h) without ever recovering. The move dispatcher operator from the dead capture to the new capture stays in the running queue for 20+ minutes, and the maintainer keeps reporting checkpointTs can not be advanced, since missing capture heartbeat.

Environment

• TiCDC: 26.3.4-1-g578cb65 (branch release-nextgen-202603)
• Architecture: new architecture
• Upstream: TiDB-X CSE cluster, PD microservices (pd / tso / scheduling split)
• Sink: downstream TiDB (sinkType=tidb), changefeed config consistent = "eventual", flush-interval = 500
• Fault injected: network partition between the PD leader and all PD followers, duration 10min (pdleader_to_pdfollower(all)), with sysbench oltp_read_write running on 64 tables upstream.

Timeline (UTC+8)

What did you expect to see

After the network partition heals and a new capture takes over, the changefeed checkpoint should resume advancing, the lag should drain back to normal (< a few seconds), and the post-failover checkpoint check should pass.

What did you see instead

The changefeed checkpoint is permanently frozen:

• Back-calculating the freeze point from the measured lag is consistent: 11:41:50 − 134.9s ≈ 11:39:35 and 14:57:21 − 11864.9s ≈ 11:39:36, i.e. the checkpoint stopped advancing at ~11:39:35 and the lag then grew linearly to ~3.3h and never recovered.
• The maintainer repeatedly logs checkpointTs can not be advanced, since missing capture heartbeat, with checkpointTs and resolvedTs both frozen at the same constant value.
• The move dispatcher operator (moving a dispatcher from the dead capture to the new capture) stays in the running queue for 20+ minutes; event_broker.go:386 dispatcher not reset is logged repeatedly.

Root cause (appears to be)

The PD leader network partition triggers a capture suicide + a nil-pointer panic in server.Close, so the old capture dies abnormally. During capture replacement in the new architecture, the dispatcher-move operator from the dead capture to the new capture gets stuck in the running queue, the dispatcher cannot complete its reset, and the maintainer therefore cannot advance checkpointTs/resolvedTs — the global checkpoint stays pinned and the lag never drains. This looks like the same family as #5553 / #5710 (changefeed can stall after capture replacement / dispatcher reset in the new architecture).

Evidence (TiCDC component logs)

[2026/09/03 11:37:54.130 +0800] Error: [CDC:ErrCaptureSuicide]capture suicide

[2026/09/03 11:39:35.776 +0800] panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20]
goroutine 860 [running]:
github.com/pingcap/ticdc/server.(*server).Close(...)
	github.com/pingcap/ticdc/server/server.go:566

[2026/09/03 11:39:37.x +0800] [maintainer.go:706] ["checkpointTs can not be advanced, since missing capture heartbeat"]
	[changefeedID=keyspace_a/ticdc-task] [checkpointTs=...] [resolvedTs=...]   (both frozen)

[2026/09/03 11:39:37.x +0800] [operator_controller.go:268] ["operator is still in running queue"]
	[operator="move dispatcher operator: ..., origin:<dead-capture>, dest:<new-capture>"]
	[timeSinceCreated=16m / 20m ... growing]

[2026/09/03 11:41:50] ChangefeedCheckpointLag check failed: checkPointLag(134.852s) > threshold(120s)
[2026/09/03 14:57:21] max checkPointLag = 11864.851s

Related

• #5553 — Changefeed can stall after CDC failover or dispatcher reset (new architecture)
• #5710 — Changefeed checkpoint can stall after capture replacement when a stale dispatcher reset fails