apache / apache/iceberg

Support Variant (and possibly other v3 types) in KafkaConnect

Open
#15,443 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.1 (latest release)

### Query engine

Kafka Connect

### Please describe the bug 🐞

I recently started to test the variant type using spark - works fine.
However, we mostly ingest data to iceberg by kafka connect, which is not able to write to a table containing variant column.
The error comes from [RecordConverter](https://github.com/apache/iceberg/blob/d2fbe427ecec298f1600260f40fcab1b7ab2e695/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java#L146).
Looking at the source code, other v3 types will be also a problem.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start at kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/data/RecordConverter.java around line 146, where conversion fails for the variant type. Compare the handling there with the other v3 types mentioned in the report. Done means Kafka Connect can write to an Iceberg table containing a variant column without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.