MySQL Sink: Recommendations for business which changes only a few rows very frequently
Open
Nobody has claimed this yet.
question
- Dominant language
- Go
- Stars
- 56
- Forks
- 63
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Before asking a question, make sure you have
- Googled your question
- Searched open and closed - TiFlow GitHub issues - TiCDC Github issues
- Read the TiCDC documentation
What is your question?
Our customer has a table with many (~30) columns. On upstream it executed UPDATE tbl SET last_updated = now() WHERE pk = 1234; very frequently with a very small set of pk. So the effect is
- event size amplification, although upstream only modified one row, downstream will need to update all 30 columns (not a very big deal), and
- non-scalable, because all UPDATE will inherently conflict with each other, these statements can never be executed in parallel and the effective worker-count is bounded by the size of the
pkset (the main problem).
The questions:
- Should TiCDC care about optimization of this scenario? Or we just have to accept this has to be slow?
- If we care, any recommended configuration to tune for downstream performance?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the TiCDC documentation linked in the issue and the MySQL sink behavior for frequent updates to a small primary-key set. Investigate whether existing downstream performance or worker-configuration guidance addresses this pattern, then document the recommended configuration or confirm that optimization would require a new design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100