pingcap / pingcap/ticdc

rebalance dispatchers by traffic when table splitting is disabled

Open
#6,251 0 comments 0 reactions 1 assignee View on GitHub

@wk989898 is already working on this.

Since Sep 14, 2026.

type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

What did you do?

With table splitting disabled, TiCDC can have equal dispatcher counts on all nodes while all replication traffic remains on one node. In a 10-minute test on master, one changefeed replicated nine tables across three TiCDC nodes, with three table dispatchers per node. We continuously wrote only to the three tables initially assigned to one node. No dispatcher migration occurred, and the other two nodes handled no DML traffic for this changefeed.

  1. Run three TiCDC nodes and create a MySQL changefeed with this configuration:
     [scheduler]
     enable-table-across-nodes = false
  1. Create 9 tables, t1 through t9, in the filtered database. Each table has a primary key, a sequence column, and a 1 KiB payload.
  2. Wait for the initial placement to reach three table dispatchers per node. Identify the three tables assigned to one node. In this run, they were t3, t5, and t7 on upstream-ticdc-0.
  3. Write only to those three tables for 10 minutes, keeping the other six tables completely idle:
    • First 180 seconds: target 1,000 row writes/s per hot table, approximately 3,000 total.
    • Next 420 seconds: target 3,000 row writes/s per hot table, approximately 9,000 total.
  4. Observe dispatcher placement and per-node sink traffic without manually moving tables or restarting nodes.
What did you expect to see?
TiCDC node Assigned tables Table dispatchers Lower-load sink throughput Higher-load sink throughput
upstream-ticdc-0 t3, t5, t7 - receiving writes 3 5.91 MiB/s 18.30 MiB/s
upstream-ticdc-1 t4, t6, t8 - idle 3 0 MiB/s 0 MiB/s
upstream-ticdc-2 t1, t2, t9 - idle 3 0 MiB/s 0 MiB/s
Image
What did you see instead?

TiCDC should detect sustained traffic imbalance and automatically migrate existing table dispatchers between nodes, even when dispatcher counts are already equal and enable-table-across-nodes = false.

For this reproduction, an expected placement would be:

TiCDC node Hot tables Idle tables Expected traffic share
upstream-ticdc-0 1 2 Approximately one third
upstream-ticdc-1 1 2 Approximately one third
upstream-ticdc-2 1 2 Approximately one third

Since the three hot tables receive equivalent workloads, this should distribute replication traffic approximately evenly while retaining three table dispatchers per node. The scheduler should account for dispatcher traffic when deciding whether a placement is balanced, preserve the no-splitting configuration, and achieve this without manual table moves. Equal dispatcher counts alone should not prevent rebalancing when their workloads differ substantially.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.