apache / apache/iceberg-python
Upsert with 1M rows extremely slow due to `create_match_filter` and `txn.delete()` performance
- 主要言語
- Python
- スター
- 1.1k
- フォーク
- 581
- 平均マージ
- 1日 17時間
- マージ済み PR(30日)
- 78
説明
### Apache Iceberg version
0.11.0
### Please describe the bug 🐞
CC @goutamvenkat-anyscale @koenvo @Fokko
Hello! We are implementing distributed writes from Ray Data to Iceberg. As part of upserts, we:
1. Write data files in parallel across Ray workers (each worker writes its share of Parquet files directly to storage and returns `DataFile` metadata + the upsert key columns back to the driver)
2. On the driver, concatenate all upsert keys collected from workers, call `create_match_filter` to build a delete predicate, then call `txn.delete()` followed by an append to commit
Upserting 1M rows (383 MiB) into an Iceberg table takes **~17.5 minutes**, almost entirely in the delete step:
```
create_match_filter (1M keys → In filter): 10.26s
txn.delete(): 1054.35s
append + commit: 1.14s
─────────────────────────────────────────────────────
Total upsert commit: 1065.75s
```
PyIceberg version `0.11.0`
This matches what's reported in #2159 and #2138.
The bottlenecks are:
1. **`create_match_filter`** — constructs a Python `BooleanExpression` node per row, which is expensive at 1M+ keys
2. **`txn.delete()`** — evaluates the resulting giant `In` expression against the table's data files with no partition pruning, effectively doing a full table scan
We have a few questions:
1. **Merge-on-read upserts** — is this on the roadmap, and if so, roughly when? MoR would let us avoid the expensive delete + rewrite cycle entirely for large upserts.
2. **Optimizing `create_match_filter` or `txn.delete()`** — is there a recommended way to speed these up today? For example, batching the `In` filter, or passing a partition-level hint to constrain the file scan?
3. **Partition-aware deletes** — if the upsert key columns overlap with partition columns, is there a supported way to restrict `txn.delete()` to only the relevant partitions, rather than scanning the full table?
## Related
- #2159 — Upserting large table extremely slow
- #2138 — Upsertion memory usage grows exponentially as table size grows
- #2943 — Optimize upsert performance for large datasets
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
レポートで説明されている create_match_filter と txn.delete() のエントリーポイントから始め、提供された計測結果をもとに 1M 行の upsert ベンチマークを再現します。既存のコンテキストについては、関連する issue #2159、#2138、#2943 を確認してください。完了条件は、測定によって確認された改善、または報告されているテーブル全体の削除コストを回避するための、文書化されたサポート対象の方法のいずれかです。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- data-engineering, databases
- issue の種類
- バグ
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 静か
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 35/100