Support Variant (and possibly other v3 types) in KafkaConnect
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.10.1 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
I recently started to test the variant type using spark - works fine.
However, we mostly ingest data to iceberg by kafka connect, which is not able to write to a table containing variant column.
The error comes from [RecordConverter](https://github.com/apache/iceberg/blob/d2fbe427ecec298f1600260f40fcab1b7ab2e695/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java#L146).
Looking at the source code, other v3 types will be also a problem.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java around line 146, where conversion fails for the variant type. Compare the handling there with the other v3 types mentioned in the report. Done means Kafka Connect can write to an Iceberg table containing a variant column without the reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100