pingcap / pingcap/ticdc

kafka: avoid duplicate topic create-and-wait operations for concurrent callers

Open
#6,083 0 comments 0 reactions 1 assignee View on GitHub

@3AceShowHand is already working on this.

Since Aug 25, 2026.

type/enhancement
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Is your feature request related to a problem?

kafkaTopicManager.GetPartitionNum checks the local topic cache and calls CreateTopicAndWaitUntilVisible on a cache miss. The cache is concurrency-safe, while the cache-miss workflow has no per-topic in-flight coordination.

A Kafka sink has independent execution paths for DML, checkpoint messages, and block events. The following pairs can occasionally access the same uncached dynamic topic concurrently:

  • DML and checkpoint
  • DDL and an earlier checkpoint operation that is still in progress
  • DML from an unrelated dispatcher and DDL when multiple tables share a dynamic topic

Each caller can independently run metadata queries, CreateTopic, and topic-visibility retries. Kafka handles concurrent creation through TopicAlreadyExists, and the Topic Ready barrier preserves correctness. The duplicate workflows still add avoidable Admin requests and metadata retries when topic visibility is slow.

Describe the feature you'd like

Ensure concurrent callers for the same topic share one in-flight CreateTopicAndWaitUntilVisible operation within a kafkaTopicManager instance.

Expected behavior:

  • The first caller starts the create-and-wait workflow.
  • Concurrent callers for the same topic wait for and reuse the same result.
  • Calls for different topics continue independently.
  • A waiting caller can return when its own context is canceled.
  • Cancellation by one waiting caller does not terminate the shared workflow for other callers.
  • A failed workflow is removed from the in-flight registry so a later call can retry.
  • A successful workflow publishes the partition count to the topic cache before releasing waiters.

Add concurrency tests covering shared success, shared failure, caller cancellation, retry after failure, and parallel operations for different topics.

Describe alternatives you've considered
  • Continue relying on Kafka TopicAlreadyExists handling and independent Topic Ready checks. This preserves correctness and retains duplicate Admin work.
  • Protect the complete cache-miss workflow with one mutex. This also serializes initialization of unrelated topics and reduces parallelism in high-topic-count changefeeds.
Teachability, Documentation, Adoption, Migration Strategy

This is an internal coordination change with no user-facing configuration or migration. Existing topic creation, partition validation, and Topic Ready semantics remain unchanged.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.