[Proposal] TiCDC Kafka Sink Retry & Fault Recovery: Short-Term and Long-Term Refinements

Background

Origin Issue: https://github.com/pingcap/tiflow/issues/12352

The new-arch TiCDC will not encounter this issue. The reason is: when the Kafka sink of the newarch CDC fails to write messages to the downstream, it does not retry at the sink level. Instead, it directly throws the error to the maintainer for handling — which is equivalent to triggering the reconstruction of the entire changefeed.

In contrast, the old-architecture CDC has this issue, and the core cause lies in the fact that its sinkManager will restart the Kafka sink internally, and there is a flaw in this part of the restart logic.

This design of the old architecture originated from Airbnb’s strict requirements on failover latency at that time: when there were occasional network fluctuations in the downstream Kafka, if the entire changefeed was restarted directly due to an error, it would trigger an incremental scan, which in turn would lead to increased latency.

Plan

Short-Term Optimization Plan

1. Add retry mechanism in Kafka Sink's AsyncSend layer
   When message sending fails, first perform limited retries in the AsyncSend phase (e.g., retry for up to 2 minutes, just as mysqlSink). If the failure persists, propagate the error to the maintainer.

   • Advantage: This balances immediate recovery and stability by handling transient network fluctuations (avoiding immediate global restarts) while eliminating potential bugs in the old architecture's sinkManager retry logic.

2. Retain current maintainer error handling logic
   Temporarily keep the new architecture's mechanism where "errors propagated to the maintainer trigger a full changefeed restart." This ensures short-term controllability and prioritizes resolving the retry logic flaws inherited from the old architecture.

Long-Term Optimization Plan

1. Refine maintainer's error handling capabilities
   Enhance the maintainer's ability to identify and localize errors, such as:

   • If only a single node's dispatcherManager is faulty, migrate the dispatcher to another healthy node instead of restarting the entire changefeed.
   • Reduce global restarts caused by local failures, lowering recovery costs and latency.

2. Leverage sorter's data retention mechanism
   Utilize the new architecture's sorter data retention to avoid incremental scans during restarts, further optimizing failover latency while enabling more flexible local scheduling.

3. Replace the old architecture's "local retry" advantages
   Combine "limited AsyncSend retries + maintainer's local scheduling" to retain the old architecture's benefit of "local node retries without global impact" while eliminating its retry logic flaws, achieving more efficient fault self-healing.

The proposal centers on the Kafka sink's AsyncSend layer, mysqlSink retry behavior, maintainer error handling, dispatcherManager scheduling, and sorter retention. Start by tracing these entry points and comparing current retry and restart behavior; done means an agreed short-term implementation and a validated long-term design, but no files or tests are named.