apache / apache/iceberg-python
Error when upserting and updating dataframes with dictionary encoded columns
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Apache Iceberg version
0.9.1
### Please describe the bug 🐞
df = pyarrow.read_table(read_dictionary=[string_columns])
table = catalog.load_table(table_name)
table.append(df) -> 'append' casts dictionary columns into large string and table appended without any error
table.upsert(df, join_cols=[primary_keys]) -> Error (Invalid Type Dictionary)
table.update(df, join_cols=[primary_keys]) -> Error (Invalid Type Dictionary)
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with pyarrow.read_table(read_dictionary=[string_columns]) and a table loaded from catalog.load_table(table_name). Compare table.append(df), table.upsert(df, join_cols=[primary_keys]), and table.update(df, join_cols=[primary_keys]); done means upsert and update handle dictionary-encoded columns without the reported Invalid Type Dictionary error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100