apache / apache/iceberg-python

Segfault on large multi-column Iceberg upserts

未关闭
#3,508 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
1.1k
派生
581
平均合并
1 天 17 小时
30 天内合并 PR
78

描述

### Apache Iceberg version

0.11.0 (latest release)

### Please describe the bug 🐞

When upserting into an Iceberg table, PyIceberg first scans the target table to
find which existing rows match the source rows' key columns. It builds that
"matching" predicate in ``pyiceberg.table.upsert_util.create_match_filter``:

* For a **single** join column it emits one flat ``In(col, [v1, v2, ...])``.
PyArrow lowers this to a single ``is_in`` compute node, no matter how many
values it contains — so single-column upserts of huge tables are fine.

* For a **multi-column** key it instead emits one disjunct per distinct key
tuple::

Or(And(c1 == v1, c2 == w1),
And(c1 == v2, c2 == w2),
...) # ONE disjunct PER ROW

PyIceberg builds that ``Or`` as a balanced tree, so the *Python* side copes.
But when the expression is handed to PyArrow's dataset scanner as a filter, the
C++ expression engine canonicalises it: ``Dataset::GetFragments`` calls
``SimplifyWithGuarantee`` → ``Canonicalize``, which flattens the associative
``or_kleene`` chain and then **recurses** over it. With tens of thousands of
disjuncts that recursion overflows the C++ call stack and the **process
segfaults** (SIGSEGV) — typically after several minutes of work, with a
backtrace full of ``arrow::compute::Canonicalize`` / ``ModifyExpression``
frames.

Reference: https://github.com/apache/iceberg-python/issues/3272

Note that apache/iceberg-python#3448 addresses a *different* upsert segfault (a
per-batch Acero re-filter in ``_task_to_record_batches``, mostly observed on
Apple Silicon). It does not touch the ``GetFragments`` canonicalisation path
exercised here, so it does not help with this crash.

The fix
-------
Produce a predicate that matches exactly the same rows, but with far fewer
disjuncts. Group the key tuples and emit a single ``In`` over whichever column
collapses to the fewest distinct "prefix" combinations (choosing that column
makes the result independent of the caller's column ordering)::

Or(And(c1 == v1, c2 IN [w, x, y]),
And(c1 == v2, c2 IN [z]),
...) # one disjunct per distinct PREFIX

The disjunct count drops from "number of rows" to "number of distinct prefix
values". In the synthetic data below there are 50 000 unique ids spread over
just 50 group values, so the predicate shrinks from 50 000 disjuncts to 50 —
shallow enough that PyArrow's canonicaliser no longer overflows.

Caveat
------
This helps whenever at least one key column is low-cardinality (or, equivalently,
one column is near-unique and can be folded into the ``In``). A genuinely
high-cardinality *composite* key — where every column is near-unique and all of
them are needed to identify a row — still produces roughly one disjunct per row
even after grouping, and can still overflow. For that pathological case the
only robust option is to upsert in smaller batches.

[pyiceberg-stacktrace.txt](https://github.com/user-attachments/files/28956590/pyiceberg-stacktrace.txt)

[iceberg_upsert_segfault_repro.py](https://github.com/user-attachments/files/28956597/iceberg_upsert_segfault_repro.py)

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 pyiceberg.table.upsert_util.create_match_filter 开始,运行 iceberg_upsert_segfault_repro.py 以复现多列 upsert 崩溃,并使用 pyiceberg-stacktrace.txt 确认规范化路径。将键元组分组为更少的析取式,同时保留精确匹配,然后验证合成的低基数场景不再发生 segfault,并审查高基数注意事项。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
data-engineering, databases
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
描述清楚
新手友好度
64/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。