apache / apache/iceberg

Iceberg integration with Spark's CDC

Open
#17,949 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

Spark 4.2 introduces native support for change data capture, functionality wise it is similar to Iceberg's create changelog view

https://spark.apache.org/releases/spark-release-4-2-0.html#change-data-capture-spark-5566,

It is good to support and even unify the semantic.

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with the linked Spark 4.2 change data capture documentation and compare its semantics with Iceberg’s create changelog view. Clarify the unified behavior and integration scope with the Iceberg community; done means Spark CDC support is implemented with matching semantics and coverage for the relevant integration behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.