apache / apache/beam

[Feature Request] : Is it possible for apache beam mongo sdk to include support to read Mongodb cdc (Change data capture) streams ?

Open
#23,551 4 comments 5 reactions 0 assignees View on GitHub
io mongodb new feature P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What would you like to happen?

Hi folks , There are many use cases where people want realtime migration of collections of Mongodb to Big query (Data Warehouse ) / GCS (Data lake). Hence , it would be great if apache beam mongo sdk included support to read Mongodb cdc (Change data capture) streams in real time and those events capture can be dumped into big query or GCS for analytical purposes. Currently to achieve this stuff we need a lot complex architecture in GCP including debezium , kafka etc. If this feature is implemented properly there will be only one single pipeline which will do all the stuff.

### Issue Priority

Priority: 3

### Issue Component

Component: io-py-mongodb

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.