apache / apache/doris-flink-connector

[Feature](mongodb-cdc) Expose split-related CDC options for MongoDB

Open
#665 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

## Problem Summary

Currently `mongodb-sync-database` does not expose several useful MongoDB CDC source options that are available in the underlying Flink CDC connector. Users cannot configure snapshot chunk granularity or reader behavior, which can lead to OOM when syncing large collections.

## Proposed Changes

Expose the following MongoDB CDC source options as configurable parameters:

- `scan.incremental.snapshot.chunk.size.mb`
- `scan.incremental.snapshot.chunk.samples`
- `split.meta.group.size`
- `close.idle.readers`
- `scan.full-changelog`
- `scan.cursor.no-timeout`

## Checklist

1. Does it affect the original behavior: No (adds new options)
2. Has unit tests been added: No
3. Has document been added or modified: No Need
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No

Contributor guide

Open the contributing guide

Research direction

Start by locating the MongoDB source option configuration used by mongodb-sync-database and compare it with the six option names listed in the issue. Add coverage for the exposed parameters and verify that the connector accepts the configured values without changing existing defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.