Kafka Connect: Add delta writer support
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Feature Request / Improvement
The initial Kafka Connect sink submission did not include delta writer support that the Tabular version has, as there are performance concerns over relying on equality deletes. We should address those concerns and add back the delta writer support, for features like change data capture and upsert mode.
### Query engine
None
### Willingness to contribute
- [X] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Read the Kafka Connect sink's current write path and compare it with the Tabular version's delta writer support. Trace how equality deletes relate to change data capture and upsert mode; done means adding equivalent delta writer support while addressing the stated performance concerns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100