coordinator: resent RunningError advances backoff multiple times during one restart
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
What did you do?
Run a changefeed and trigger a retryable error in a DispatcherManager.
After receiving its first error, DispatcherManager sends the corresponding RunningError immediately and resends the same message every five seconds until the dispatcher manager is closed:
// resend message until the event dispatcher manager is closed
ticker := time.NewTicker(time.Second * 5)
These repeated reports can reach the coordinator while the same changefeed restart is still in progress.
What did you expect to see?
Repeated delivery of the same RunningError during one restart attempt should advance the backoff once.
A later failed restart attempt may advance the next backoff step.
What did you see instead?
Each resent RunningError reaches Backoff.CheckStatus.
When isRestarting is already true, CheckStatus logs that the error is ignored, then continues into HandleError. Each call to HandleError invokes NextBackOff and updates nextRetryTime.
One observed sequence was:
10:59:33.189 nextRetryTime = 11:02:14
10:59:33.190 already in restarting progress
10:59:33.190 nextRetryTime = 11:05:18
10:59:33.190 already in restarting progress
10:59:33.190 nextRetryTime = 11:08:45
All three reports contained the same error timestamp, source node, error code, and message.
A single failed restart attempt can therefore consume multiple exponential backoff steps, delaying recovery and potentially exhausting the configured backoff duration prematurely.
Versions of the cluster
Upstream TiDB cluster version:
N/A — this behavior is independent of the upstream TiDB version.
Upstream TiKV version:
N/A — this behavior is independent of the upstream TiKV version.
TiCDC version:
v26.3.4
Git commit: 65002945d670fea61492e35545326a0a289853c5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing DispatcherManager's RunningError resend path into the coordinator's Backoff.CheckStatus, HandleError, and NextBackOff flow, then reproduce the changefeed restart scenario described in the issue. Done means repeated identical RunningError reports during one restart advance backoff only once, while a later failed restart advances the next step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100