debezium / debezium/dbz

MongoDB Transformation Stage [DBZ-5995]

Open
#741 0 comments 0 reactions 0 assignees View on GitHub
component/mongodb-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-5995](https://issues.redhat.com/browse/DBZ-5995)

Currently, I am using the MongoDB Kafka Source Connector to stream the event in my local database to another MongoDB instance in the cloud.  We want to replace the remote DB by a Postgresql instance and to do it, we want to use the Debezium Mongodb source connector with the Debezium sink connector for Postgresql.

My source connector has an aggregation pipeline to modify the payload to another schema before putting it into Kafka. The example below shows how to achieve such behavior:

```
{
  "name": "metrics",
  "config": {
    "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "connection.uri": "mongodb://172.0.0.2:27017",
    "database": "metrics",
    "collection": "metrics",
    "copy.existing": "true",
    "pipeline": "[\{"$match": { "$and": [{"operationType": "insert"}, \{"fullDocument.eventId": 1404 }] } }]"
  }
}
```

It would be great to Debezium source connector for MongoDB support this kind of aggregation pipeline to transform each streamed document. In another words, The feature consists in a SMT using mongoDB aggregation stage.

[MongodDB Documentation Example|mongodb.com/docs/kafka-connector/current/source-connector/configuration-properties/all-properties/#change-streams]

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with the MongoDB Kafka Source Connector configuration and the aggregation pipeline shown in the issue. Trace how streamed documents enter Kafka, then identify the connector entry point for an SMT; done means the source connector supports a MongoDB aggregation stage that transforms each streamed document.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, mongodb, postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.