apache / apache/iceberg

Kafka Connect: Add delta writer support

Open
#10,842 13 comments 13 reactions 0 assignees View on GitHub
improvement KAFKACONNECT
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

The initial Kafka Connect sink submission did not include delta writer support that the Tabular version has, as there are performance concerns over relying on equality deletes. We should address those concerns and add back the delta writer support, for features like change data capture and upsert mode.

### Query engine

None

### Willingness to contribute

- [X] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Read the Kafka Connect sink's current write path and compare it with the Tabular version's delta writer support. Trace how equality deletes relate to change data capture and upsert mode; done means adding equivalent delta writer support while addressing the stated performance concerns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.