Unbounded Source for MongoDB
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
The current MongoDB source is bounded, which means that we can't build streaming pipelines directly from MongoDB.
MongoDB publishes changes in each collection through the oplog. Would it be possible to create a connector that reads the oplog to create an unbounded source?
As an oplog is only available through replication, this creates that dependency. We would need to also consider whether a polling method (using the ObjectId) could be an appropriate fallback.
Thanks
Imported from Jira [BEAM-2639](https://issues.apache.org/jira/browse/BEAM-2639). Original Jira may contain additional context.
Reported by: nevi_me.
Contributor guide
Research direction
Start by reviewing the existing MongoDB source and the oplog and replication requirements described in the issue. Determine whether an unbounded oplog-based connector or an ObjectId polling fallback is in scope; done requires a defined approach and a streaming source implementation with corresponding validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100