new architecture: traffic balancing can move spans to an overloaded EventStore node

Summary

In TiCDC v8.5.7 new architecture, traffic balancing moved four table spans to a node that was already handling a large historical EventStore catch-up. The scheduler selected the destination from group-local, sink-flushed output traffic; it could not see node-wide EventStore input or initialization pressure from another changefeed. All four move operators then remained running and blocked watermark progress for the affected changefeeds.

This report uses stable neutral aliases for every node, changefeed, table, dispatcher, and cluster identifier:

• Node-T: overloaded move destination
• Node-A / Node-B: move sources
• CF-A / CF-B: existing changefeeds with split-table dispatchers
• CF-C: newly initialized changefeed with an old start-ts
• table-A / table-B / table-C: affected tables

The monitoring chart below is a sanitized reconstruction from the original 15-second samples. It contains no original node, table, changefeed, organization, or cluster names.

Incident timeline

All times are UTC on 2026-08-28.

Time	Event
04:00:40.073	Physical time encoded in CF-C's start-ts. This is the replication start position, not the dispatcher creation time.
19:45:41.728–19:45:41.754	CF-C dispatcher managers initialized. Its start-ts was already about 15 h 45 min old.
~19:45:42	Two recovered auxiliary dispatchers were created on peer nodes. The corresponding table-C creation line on Node-T is absent from the collected logs, but the peer timing and dashboard constrain its creation to 19:45:41.7–19:45:45.
19:45:45	Node-T had 26 event dispatchers: 16 from CF-A/table-A, 9 from CF-B/table-B, and 1 from CF-C/table-C. EventStore input was 27.95 MiB/s.
19:46:00 / 19:46:15	Before any move, Node-T EventStore input reached 150.90 MiB/s. The directly isolated increase over the prior sample was 122.95 MiB/s and coincided with the new CF-C/table-C historical subscription.
19:46:23.734	CF-A scheduled the first move batch: three table-A spans from Node-A to Node-T.
19:46:23.835–19:46:23.884	The three source dispatchers were stopped/unregistered and rebound to Node-T; its dispatcher count increased from 26 to 29.
19:46:24.437	Destination ADD/registration processing had started. The observed old-source onlyReuse=true probe failures are expected when EventStore data sharing is disabled and are not themselves the failure.
19:46:30 / 19:46:45 / 19:47:00	Node-T EventStore input rose to 243.59 / 351.42 / 431.09 MiB/s as the one-minute rate window filled and the new subscriptions ran.
19:46:53.850	The three CF-A moves emitted their first 30-second “still in running queue” warnings; the destination had not reported Working.
19:47:48.081	CF-B scheduled another move: one table-B span from Node-B to Node-T.
19:47:48.188–19:47:48.267	The source was stopped and rebound to Node-T; its dispatcher count increased from 29 to 30.
19:48:18.242	The CF-B move emitted its first 30-second running warning.
19:48:45	Node-T EventStore input peaked at 451.80 MiB/s while peer nodes stayed in an observed 19–31 MiB/s band.
20:14:49–20:14:55	The CF-B move had been running for about 27 minutes and the three CF-A moves for about 28.5 minutes.
20:15:00	The collected log window ended with all four move operators still running and Node-T holding 30 dispatchers.

The node-level metric has no dispatcher label in v8.5.7, so the final 451.80 MiB/s peak cannot be decomposed exactly. The pre-move 122.95 MiB/s increase is directly isolated to the newly registered CF-C/table-C subscription. After the one-minute rate window filled, its sustained historical-scan input is estimated at roughly 400–425 MiB/s, but the four later subscriptions can contribute an unknown portion of that plateau.

What happened

The scheduler's traffic score is based on EventSizePerSecond for spans in the current split-table group. That value is updated from events flushed by the sink. It is not a node-wide capacity signal and does not include:

• EventStore/logpuller input from another changefeed;
• historical catch-up or initializing subscriptions;
• EventStore write backlog or memory pressure;
• destination CPU, network, sink latency, or initialization readiness.

Consequently, CF-A and CF-B both considered Node-T a low-output destination even though another changefeed was already driving very high EventStore input there.

The move path removed each source first and completed only after the destination reported Working. The four destination dispatchers did not reach Working during the captured interval. The 30-second age check only emitted warnings; it did not time out, clean up, retry elsewhere, or exclude the destination. These running move operators continued to block watermark forwarding.

The evidence boundary is:

• Proven: Node-T had extreme EventStore input before the first move; the placement score could not see it; four source-first moves selected that node; none completed during the remaining log window.
• Strongly inferred: the historical subscription and additional initialization work put the local EventStore/logpuller path under enough pressure that the destination dispatchers did not reach Working promptly.
• Not directly observable: the exact target-local stall stage, because the available archive does not contain the Node-T process log and v8.5.7 lacks per-subscription input/progress metrics.

Expected behavior

• Placement should not select a destination that is already overloaded by node-wide EventStore/logpuller work from another changefeed.
• A move that cannot make destination-side progress should have a safe recovery path and must not block watermark forwarding indefinitely.
• Initialization pressure and move progress should be observable through metrics/API.

Reversed configuration comments

The comments for MinTrafficPercentage and MaxTrafficPercentage in pkg/config/scheduler_config.go describe the effect in the wrong direction.

These two parameters do not directly control a time frequency. They define how far a node's traffic must deviate from the per-sample average before the sample is considered unbalanced. The runtime first computes:

avgTraffic = totalTraffic / nodeCount

It then checks the most recent three samples. The implementation in checkBalanceTraffic is equivalent to:

lowTrafficUnbalanced :=
    minNodeTraffic <= avgTraffic * minTrafficPercentage

highTrafficUnbalanced :=
    maxNodeTraffic >= avgTraffic * maxTrafficPercentage

shouldBalance := lowTrafficUnbalanced || highTrafficUnbalanced

The selected condition must hold for all three recent samples before the balance score is incremented.

Parameter	Unbalanced condition over the last three samples	Actual effect of changing it
MinTrafficPercentage	Lowest node: traffic <= avg * min in all three samples	Larger value makes the condition easier to satisfy, so balancing becomes more frequent
MaxTrafficPercentage	Highest node: traffic >= avg * max in all three samples	Smaller value makes the condition easier to satisfy, so balancing becomes more frequent

The defaults are:

min-traffic-percentage = 0.8
max-traffic-percentage = 1.25

That means traffic is considered unbalanced when the lowest node stays at or below 80% of the average, or the highest node stays at or above 125% of the average.

For example, with a stable average of 100 MiB/s:

• Minimum-node traffic 85, 85, 85: min=0.8 gives a lower boundary of 80 and does not trigger; min=0.9 gives a boundary of 90 and does trigger. Increasing min therefore makes balancing more frequent.
• Maximum-node traffic 115, 115, 115: max=1.25 gives an upper boundary of 125 and does not trigger; max=1.1 gives a boundary of 110 and does trigger. Decreasing max therefore makes balancing more frequent.

The comments should instead say, for example:

// MinTrafficPercentage:
// Larger value means more frequent balancing.

// MaxTrafficPercentage:
// Smaller value means more frequent balancing.

BalanceScoreThreshold is the separate knob that requires the unbalanced condition to be observed repeatedly before a move is scheduled.

Version

• TiCDC: v8.5.7, new architecture