apache / apache/doris-flink-connector
[Feature](mongodb-cdc) Expose split-related CDC options for MongoDB
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
## Problem Summary
Currently `mongodb-sync-database` does not expose several useful MongoDB CDC source options that are available in the underlying Flink CDC connector. Users cannot configure snapshot chunk granularity or reader behavior, which can lead to OOM when syncing large collections.
## Proposed Changes
Expose the following MongoDB CDC source options as configurable parameters:
- `scan.incremental.snapshot.chunk.size.mb`
- `scan.incremental.snapshot.chunk.samples`
- `split.meta.group.size`
- `close.idle.readers`
- `scan.full-changelog`
- `scan.cursor.no-timeout`
## Checklist
1. Does it affect the original behavior: No (adds new options)
2. Has unit tests been added: No
3. Has document been added or modified: No Need
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No
Contributor guide
Research direction
Start by locating the MongoDB source option configuration used by mongodb-sync-database and compare it with the six option names listed in the issue. Add coverage for the exposed parameters and verify that the connector accepts the configured values without changing existing defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100