apache / apache/iceberg-python
BucketTransform fails on dictionary-encoded pyArrow arrays
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Apache Iceberg version
0.9.1
### Please describe the bug 🐞
Pyiceberg version - 0.9.1 (also reproduced with 0,11.1)
Pyiceberg core version - 0.9.1
table created with bucket transform
df = pyarrow.read_table(read_dictionary=string_columns)
table.append(df)
ValueError : Feature Unsupported => Unsupported data type for bucket transform: Dictionary (Int32, Utf8)
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with pyarrow.read_table(read_dictionary=string_columns) followed by table.append(df) using a bucket transform. Inspect the bucket-transform handling for Dictionary (Int32, Utf8) arrays; done means dictionary-encoded columns can be appended without the unsupported-data-type error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100