Repeated runtime state persistence can put excessive load on PD/etcd when many changefeeds enter warning

Summary

When many changefeeds are stuck and repeatedly report runtime errors, TiCDC repeatedly persists runtime state by rewriting the whole changefeed/info value in etcd. ChangeFeedInfo contains the full replica config and other metadata, so these updates become large OpPut operations. If many changefeeds enter warning at the same time, this creates write amplification and can put excessive load on PD/etcd.

Why this is not only a warning problem

This is not a warning-only bug. The same persistence path is also used for failed, finished, and recovery back to normal, and stale checkpoint GC checks can also trigger it. warning is just the most visible case because it can be generated repeatedly while the checkpoint does not advance.

Root cause

1. When checkpointTs does not advance and errors keep being reported, Backoff.CheckStatus keeps returning changed=true and StateWarning for retryable errors.
2. The controller wraps the reported error into a new RunningError with Time: time.Now().
3. The coordinator stores state and error back into ChangeFeedInfo.
4. UpdateChangefeed marshals and rewrites the entire ChangeFeedInfo to etcd with OpPut.
5. Because ChangeFeedInfo is large, repeated runtime state updates become repeated large etcd writes.

Relevant code paths include coordinator/changefeed/backoff.go, coordinator/controller.go, coordinator/coordinator.go, and coordinator/changefeed/etcd_backend.go.

Impact

• Large etcd value rewrites during warning storms
• Larger txn payloads and write amplification
• Excessive load on PD/etcd when many changefeeds hit the same pattern

Proposed solution

Short term

• Deduplicate runtime state persistence before calling UpdateChangefeed.
• Only persist when the semantic state actually changes.
• Compare state, error.Code, error.Addr, and error.Message, but ignore RunningError.Time.
• Keep checkpoint-only updates on the existing lightweight status path.

Optional safeguard

• Add stronger fast-fail rules for clearly unrecoverable errors.
• Optionally fast-fail when the same error signature repeats for a long time and checkpointTs does not move, to put an upper bound on the warning-retry loop.
• This can reduce repeated writes, but it is only a mitigation and should not replace deduplication.

Long term

• Split runtime state and runtime error from ChangeFeedInfo.
• Keep static config and metadata in ChangeFeedInfo, and persist runtime status in a lightweight key or object.
• Then warning / failed / finished / recovery updates no longer need to rewrite the full metadata object.

Recommendation

The recommended order is:

1. Add state persistence deduplication first.
2. Add targeted fast-fail rules if needed for long-lived repeated warnings.
3. Long term, separate runtime state from ChangeFeedInfo.