Iceberg integration with Spark's CDC
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Feature Request / Improvement
Spark 4.2 introduces native support for change data capture, functionality wise it is similar to Iceberg's create changelog view
https://spark.apache.org/releases/spark-release-4-2-0.html#change-data-capture-spark-5566,
It is good to support and even unify the semantic.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start with the linked Spark 4.2 change data capture documentation and compare its semantics with Iceberg’s create changelog view. Clarify the unified behavior and integration scope with the Iceberg community; done means Spark CDC support is implemented with matching semantics and coverage for the relevant integration behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100