apache / apache/iceberg-python
Upserting large table extremely slow
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Feature Request / Improvement
## Feature Request / Improvement
Upserting large dataframes (tens of millions of rows) in un-usably slow due to creating a massive `BooleanExpression` in `upsert_util.create_match_filter`. This is before any IO is even started. It would be nice if upserts could support large tables.
I'd be happy to work on this issue.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at upsert_util.create_match_filter, where the issue identifies a massive BooleanExpression as the pre-I/O bottleneck. Trace how this filter is used during dataframe upserts and measure behavior with a large table. Done means upserting tens of millions of rows no longer spends an unusable amount of time constructing the match filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100