Enhance mqSink to support Low-latency/DDL-consistent/Txn-consistent modes
Open
@liuzix is already working on this.
Since Jul 27, 2020.
component/sink
difficulty/hard
- Dominant language
- Go
- Stars
- 464
- Forks
- 313
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 14
Description
Feature Request
Is your feature request related to a problem? Please describe:
- Canal support is not DDL-consistent in the sense that a DDL might be reordered with DMLs.
- In order to make Canal support DDL-consistent, we sacrifice latency because we need to wait until
FlushRowChangedEventsis issued. - If we write transaction reconstruction logic in mqSink, code complexity will be increased, and because different encoders might want to handle transactions differently, letting mqSink construct the transactions hampers the decoupling of code.
Describe the feature you'd like:
- DDL-consistent mode should be achieved by preventing the mqProducer from implicit flushes, i.e. write to the MQ without explicit
Flushcalls. Hence I propose writing a mqProducer decorator that can make any producer hold back sending data to network untilFlushis called. - Txn-consistent mode should be supported directly by encoders, which should reconstruct the transactions internally. To this end, we should provide
TxnGeneratorthat facilitates the reconstruction of transactions.TxnGeneratorshould supportAppendandSplitmethods. Changes in #770 should be merged so that the encoder can control when it wants the sink to write to the MQ.
Current support/needs
| Open Protocol | Canal | Avro | |
|---|---|---|---|
| Low-latency | Available | Available | Available |
| DDL-consistent | No plan | Needed to ensure correctness | No need because there's no DDL output |
| Transaction-consistent | Handled by consumer | Planned | No need because there's no semantics |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.