pingcap / pingcap/ticdc

MySQL Sink: Recommendations for business which changes only a few rows very frequently

Open
#1,646 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Go
Stars
56
Forks
63
Avg merge
2d 20h
Merged PRs (30d)
34

Description

Before asking a question, make sure you have
What is your question?

Our customer has a table with many (~30) columns. On upstream it executed UPDATE tbl SET last_updated = now() WHERE pk = 1234; very frequently with a very small set of pk. So the effect is

  1. event size amplification, although upstream only modified one row, downstream will need to update all 30 columns (not a very big deal), and
  2. non-scalable, because all UPDATE will inherently conflict with each other, these statements can never be executed in parallel and the effective worker-count is bounded by the size of the pk set (the main problem).

The questions:

  • Should TiCDC care about optimization of this scenario? Or we just have to accept this has to be slow?
  • If we care, any recommended configuration to tune for downstream performance?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the TiCDC documentation linked in the issue and the MySQL sink behavior for frequent updates to a small primary-key set. Investigate whether existing downstream performance or worker-configuration guidance addresses this pattern, then document the recommended configuration or confirm that optimization would require a new design.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.