Multi-task setup [DBZ-6984]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-6984](https://issues.redhat.com/browse/DBZ-6984)
During our load test, we found that currently Debezium mongo connector is capped around 30MB/s-40MB/s due to it's single threaded way of streaming.
Currently although each connector can work on different DB/table, but it's difficult to solve the hot table issue. With the proposed setup, this can be scaled up easily.
Currently we are planning to do a multi-task setup for mongodb, i.e. three connectors reading from a single replset. Each connector only works 1/3 of the traffic split by hash of time. With this, Debezium throughput can be improved to 100+MB/s.
I'm not sure if
* Debezium community will be interested in this patch for mongo, we will be happy to send this patch out once it's done.
* other databases could apply the similar approach?
Or is there any other solution you have already tried?
Contributor guide
Research direction
Start by reviewing the Debezium MongoDB connector's current single-threaded streaming behavior and the proposed multi-task setup described in the issue. The work is ready when the community agrees on a concrete design for splitting replset traffic across tasks and defines how the approach should be validated against the stated throughput goal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100