Prevent frequent schema changes in MongoDB connector using Avro converter [DBZ-1154]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-1154](https://issues.redhat.com/browse/DBZ-1154)
When using MongoDB connector with Avro converter, and UnwrapFromMongoDbEnvelope, schema changes at a very frequent pace, the result is Too many schemas created error from schema-registry, or a slow down in Kafka-connect worker.
There are 2 reasons for this behavior:
1. Frequent schema changes in a mongoDB collection.
2. Changes in field order sent to schema-registry.
Proposed solution is to cache schema per topic, and add new fields to it as optional fields so schema will evolve to contain all fields, this will be done while keeping the order of the fields.
Contributor guide
Research direction
Start with the MongoDB connector path involving UnwrapFromMongoDbEnvelope and the Avro converter described in the issue. Trace how schemas are sent to Schema Registry, then verify that frequent changes and field-order changes no longer create excessive schemas while new fields remain optional and ordered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mongodb
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100