pingcap / pingcap/ticdc

[P1][Logs] Collapse merge, close, and retry loop logs into summaries

Open
#4,738 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Parent issue: #4691

Intent

High-frequency merge, close, and retry loops can emit repeated Info logs every few milliseconds. In large or stuck workflows this creates log storms. This task preserves the first, timeout, and terminal logs while collapsing the middle of the loop into structured summaries.

Modification points

  • Convert repeated loop logs such as still not closed, waiting for dispatchers to be closed, and retry progress logs into window-based summaries.
  • Keep counters such as retryCount, closedCount, and pendingCount in the summary instead of logging every iteration.
  • Preserve only logs that mark the first occurrence, timeout escalation, or final result.

Entry points

  • downstreamadapter/dispatchermanager/task.go:281
  • downstreamadapter/dispatchermanager/task.go:286
  • downstreamadapter/dispatchermanager/task.go:525
  • downstreamadapter/dispatchermanager/task.go:536
  • downstreamadapter/dispatcher/basic_dispatcher.go:1246

Done criteria

  • A long-running merge or close operation no longer emits repeated loop logs linearly with retry count.
  • The retained logs still show progress, duration, and pending work clearly enough for diagnosis.
  • Operators can distinguish first occurrence, ongoing retry, timeout, and final completion from the remaining logs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the loop entry points in downstreamadapter/dispatchermanager/task.go at lines 281, 286, 525, and 536, then inspect downstreamadapter/dispatcher/basic_dispatcher.go:1246. Trace the merge, close, and retry logging paths and identify which repeated messages can become window-based summaries while retaining counters and terminal events. Done means long-running operations avoid linear log growth while logs still distinguish progress, timeout, pending work, and completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.