debezium / debezium/dbz

Prevent frequent schema changes in MongoDB connector using Avro converter [DBZ-1154]

Open
#207 0 comments 0 reactions 0 assignees View on GitHub
component/mongodb-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-1154](https://issues.redhat.com/browse/DBZ-1154)

When using MongoDB connector with Avro converter, and UnwrapFromMongoDbEnvelope, schema changes at a very frequent pace, the result is Too many schemas created error from schema-registry, or a slow down in Kafka-connect worker.

There are 2 reasons for this behavior:
1. Frequent schema changes in a mongoDB collection.
2. Changes in field order sent to schema-registry.

Proposed solution is to cache schema per topic, and add new fields to it as optional fields so schema will evolve to contain all fields, this will be done while keeping the order of the fields.

Contributor guide

Open the contributing guide

Research direction

Start with the MongoDB connector path involving UnwrapFromMongoDbEnvelope and the Avro converter described in the issue. Trace how schemas are sent to Schema Registry, then verify that frequent changes and field-order changes no longer create excessive schemas while new fields remain optional and ordered.

Written by the indexing model from the issue text.

Assessment

Tech stack
mongodb
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.