MongoDB Transformation Stage [DBZ-5995]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-5995](https://issues.redhat.com/browse/DBZ-5995)
Currently, I am using the MongoDB Kafka Source Connector to stream the event in my local database to another MongoDB instance in the cloud. We want to replace the remote DB by a Postgresql instance and to do it, we want to use the Debezium Mongodb source connector with the Debezium sink connector for Postgresql.
My source connector has an aggregation pipeline to modify the payload to another schema before putting it into Kafka. The example below shows how to achieve such behavior:
```
{
"name": "metrics",
"config": {
"connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
"connection.uri": "mongodb://172.0.0.2:27017",
"database": "metrics",
"collection": "metrics",
"copy.existing": "true",
"pipeline": "[\{"$match": { "$and": [{"operationType": "insert"}, \{"fullDocument.eventId": 1404 }] } }]"
}
}
```
It would be great to Debezium source connector for MongoDB support this kind of aggregation pipeline to transform each streamed document. In another words, The feature consists in a SMT using mongoDB aggregation stage.
[MongodDB Documentation Example|mongodb.com/docs/kafka-connector/current/source-connector/configuration-properties/all-properties/#change-streams]
Thanks
Contributor guide
Research direction
Start with the MongoDB Kafka Source Connector configuration and the aggregation pipeline shown in the issue. Trace how streamed documents enter Kafka, then identify the connector entry point for an SMT; done means the source connector supports a MongoDB aggregation stage that transforms each streamed document.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, mongodb, postgresql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100