pingcap / pingcap/tiflow

Enhance mqSink to support Low-latency/DDL-consistent/Txn-consistent modes

Open
#795 2 comments 2 reactions 1 assignee View on GitHub

@liuzix is already working on this.

Since Jul 27, 2020.

component/sink difficulty/hard
Dominant language
Go
Stars
464
Forks
313
Avg merge
4d 7h
Merged PRs (30d)
14

Description

Feature Request

Is your feature request related to a problem? Please describe:

  • Canal support is not DDL-consistent in the sense that a DDL might be reordered with DMLs.
  • In order to make Canal support DDL-consistent, we sacrifice latency because we need to wait until FlushRowChangedEvents is issued.
  • If we write transaction reconstruction logic in mqSink, code complexity will be increased, and because different encoders might want to handle transactions differently, letting mqSink construct the transactions hampers the decoupling of code.

Describe the feature you'd like:

  • DDL-consistent mode should be achieved by preventing the mqProducer from implicit flushes, i.e. write to the MQ without explicit Flush calls. Hence I propose writing a mqProducer decorator that can make any producer hold back sending data to network until Flush is called.
  • Txn-consistent mode should be supported directly by encoders, which should reconstruct the transactions internally. To this end, we should provide TxnGenerator that facilitates the reconstruction of transactions. TxnGenerator should support Append and Split methods. Changes in #770 should be merged so that the encoder can control when it wants the sink to write to the MQ.

Current support/needs

Open Protocol Canal Avro
Low-latency Available Available Available
DDL-consistent No plan Needed to ensure correctness No need because there's no DDL output
Transaction-consistent Handled by consumer Planned No need because there's no semantics

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.