apache / apache/iceberg-python

Error when upserting and updating dataframes with dictionary encoded columns

Open
#3,835 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Apache Iceberg version

0.9.1

### Please describe the bug 🐞

df = pyarrow.read_table(read_dictionary=[string_columns])

table = catalog.load_table(table_name)

table.append(df) -> 'append' casts dictionary columns into large string and table appended without any error

table.upsert(df, join_cols=[primary_keys]) -> Error (Invalid Type Dictionary)
table.update(df, join_cols=[primary_keys]) -> Error (Invalid Type Dictionary)

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with pyarrow.read_table(read_dictionary=[string_columns]) and a table loaded from catalog.load_table(table_name). Compare table.append(df), table.upsert(df, join_cols=[primary_keys]), and table.update(df, join_cols=[primary_keys]); done means upsert and update handle dictionary-encoded columns without the reported Invalid Type Dictionary error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.