apache / apache/iceberg-python
Upserting large table extremely slow
Open
stale
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Feature Request / Improvement
## Feature Request / Improvement
Upserting large dataframes (tens of millions of rows) in un-usably slow due to creating a massive `BooleanExpression` in `upsert_util.create_match_filter`. This is before any IO is even started. It would be nice if upserts could support large tables.
I'd be happy to work on this issue.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.