debezium / debezium/dbz

Multi-task setup [DBZ-6984]

Open
#897 0 comments 0 reactions 0 assignees View on GitHub
component/mongodb-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-6984](https://issues.redhat.com/browse/DBZ-6984)

During our load test, we found that currently Debezium mongo connector is capped around 30MB/s-40MB/s due to it's single threaded way of streaming. 

Currently although each connector can work on different DB/table, but it's difficult to solve the hot table issue. With the proposed setup, this can be scaled up easily.

Currently we are planning to do a multi-task setup for mongodb, i.e. three connectors reading from a single replset. Each connector only works 1/3 of the traffic split by hash of time. With this, Debezium throughput can be improved to 100+MB/s.

I'm not sure if
* Debezium community will be interested in this patch for mongo, we will be happy to send this patch out once it's done.
* other databases could apply the similar approach?

Or is there any other solution you have already tried?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Debezium MongoDB connector's current single-threaded streaming behavior and the proposed multi-task setup described in the issue. The work is ready when the community agrees on a concrete design for splitting replset traffic across tasks and defines how the approach should be validated against the stated throughput goal.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.