pingcap / pingcap/ticdc

coordinator: resent RunningError advances backoff multiple times during one restart

Open
#6,082 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

What did you do?

Run a changefeed and trigger a retryable error in a DispatcherManager.

After receiving its first error, DispatcherManager sends the corresponding RunningError immediately and resends the same message every five seconds until the dispatcher manager is closed:

// resend message until the event dispatcher manager is closed
ticker := time.NewTicker(time.Second * 5)

These repeated reports can reach the coordinator while the same changefeed restart is still in progress.

What did you expect to see?

Repeated delivery of the same RunningError during one restart attempt should advance the backoff once.

A later failed restart attempt may advance the next backoff step.

What did you see instead?

Each resent RunningError reaches Backoff.CheckStatus.

When isRestarting is already true, CheckStatus logs that the error is ignored, then continues into HandleError. Each call to HandleError invokes NextBackOff and updates nextRetryTime.

One observed sequence was:

10:59:33.189  nextRetryTime = 11:02:14
10:59:33.190  already in restarting progress
10:59:33.190  nextRetryTime = 11:05:18
10:59:33.190  already in restarting progress
10:59:33.190  nextRetryTime = 11:08:45

All three reports contained the same error timestamp, source node, error code, and message.

A single failed restart attempt can therefore consume multiple exponential backoff steps, delaying recovery and potentially exhausting the configured backoff duration prematurely.

Versions of the cluster

Upstream TiDB cluster version:

N/A — this behavior is independent of the upstream TiDB version.

Upstream TiKV version:

N/A — this behavior is independent of the upstream TiKV version.

TiCDC version:

v26.3.4
Git commit: 65002945d670fea61492e35545326a0a289853c5

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing DispatcherManager's RunningError resend path into the coordinator's Backoff.CheckStatus, HandleError, and NextBackOff flow, then reproduce the changefeed restart scenario described in the issue. Done means repeated identical RunningError reports during one restart advance backoff only once, while a later failed restart advances the next step.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.