apache / apache/datafusion-python
Segfault with PyArrow 24 on macOS when using mimalloc v3
- 主要語言
- Python
- 星號
- 604
- 分支
- 174
- 平均合併
- 2 天 22 小時
- 30 天內合併 PR
- 5
描述
## Describe the bug
After the DataFusion 54 upgrade (#1562), importing `datafusion` and performing any Arrow-backed operation segfaults (SIGSEGV) when the installed PyArrow is 24.0.0 on macOS (arm64). The crash happens on the very first Arrow allocation made through the bindings — for example building a literal `lit(pa.scalar(0, type=pa.int32()))`, which is exactly what `python/datafusion/functions/spark.py` does at module import, so even a bare `import datafusion` crashes.
This is a regression introduced on the 54 upgrade branch; it does **not** affect the released `datafusion-python` 53.0.0.
## Symptoms
`import datafusion` (or any operation that constructs an Arrow value) terminates the process with `Segmentation fault: 11` (exit code 139). The native crash report points into PyArrow's own bundled mimalloc, not into our code:
```
mi_theap_malloc_zero_aligned_at_overalloc <- SIGSEGV (mimalloc v3 thread-heap)
mi_theap_realloc_zero_aligned_at
arrow::MimallocAllocator::ReallocateAligned
arrow::PoolBuffer::Resize
arrow::NumericBuilder::FinishInternal
arrow::py::ConvertPySequence
__pyx_pw_7pyarrow_3lib_191scalar <- pa.scalar(0, type=int32())
```
## Root cause
There are two independent mimalloc runtimes in the process:
- `datafusion-python` installs mimalloc as the Rust `#[global_allocator]` (`crates/core/src/lib.rs`, enabled by the default `mimalloc` feature).
- PyArrow 24 ships and defaults to its own bundled mimalloc memory pool.
The DataFusion 54 dependency bump moved `libmimalloc-sys` 0.1.44 -> 0.1.49 (the `mimalloc` crate 0.1.48 -> 0.1.52), which changed the bundled allocator from mimalloc **v2** to mimalloc **v3**. PyArrow 24 also bundles mimalloc **v3**. Two mimalloc-v3 runtimes collide at the macOS process-global level (malloc-zone / thread-local-heap initialization), corrupting each other's thread heap and faulting on the first allocation.
The 53.0.0 release shipped mimalloc **v2** (`libmimalloc-sys` 0.1.44), which coexists fine with PyArrow's v3 pool — which is why no released version is affected.
## Affected versions / platforms
- **PyArrow**: 24.0.0 triggers it. PyArrow 20.0.0 through 23.0.1 are unaffected (verified against the 54-branch build).
- **datafusion-python**: the in-progress 54 upgrade branch. Released 53.0.0 is **not** affected (verified with PyArrow 20–24).
- **Platforms**: confirmed on macOS arm64. Linux is expected to be unaffected because PyArrow defaults to jemalloc there (only one mimalloc in the process). Windows defaults to mimalloc like macOS, so it is potentially affected, but the macOS-specific malloc-zone vector may not apply — needs verification in CI.
## Reproduction
On macOS arm64 with a 54-branch build of `datafusion-python` and `pyarrow==24.0.0`, remove the `"v2"` feature flag on mimalloc in `Cargo.toml`.
```python
import datafusion # segfaults here (spark.py builds an int32 literal at import)
```
or, isolating the allocation:
```python
import pyarrow as pa
from datafusion import lit
lit(pa.scalar(0, type=pa.int32())) # SIGSEGV
```
## Suggested fix
Work around to be introduced for releasing 54.0.0: Pin the bundled allocator to the mimalloc v2 line so two mimalloc-v3 runtimes never coexist. `libmimalloc-sys` (and the `mimalloc` crate) expose a `v2` feature for this; adding it to the `mimalloc` feature list in `crates/core/Cargo.toml` keeps the Rust global allocator (no performance loss, no PyArrow pin) and resolves the crash. This has been verified locally: with the `v2` feature the 54-branch build runs cleanly against PyArrow 24.0.0.
A longer-term fix should investigate making two mimalloc-v3 instances coexist (or platform-gating the allocator), and we should add a CI smoke test that imports `datafusion` and constructs an Arrow literal against the newest PyArrow on macOS so this regression cannot return silently.
## Acceptance / testing
The fix must include test coverage: a smoke test (run on macOS, and ideally Windows) that imports `datafusion` and builds an Arrow-backed literal under the newest supported PyArrow, asserting no crash.
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從 crates/core/Cargo.toml 中的 mimalloc feature 設定以及透過 python/datafusion/functions/spark.py 參照的 Arrow 配置路徑開始。使用 PyArrow 24 在 macOS 上重現崩潰,接著驗證配置器設定,並為匯入 datafusion 和建構由 Arrow 支援的常值新增所要求的 smoke coverage。完成的標準是 54-branch build 在支援的平台上不再發生 segfault,且該回歸已在 CI 中涵蓋。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python, rust
- 領域
- backend, build-system, operating-systems, testing-qa
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 描述清楚
- 新手友好度
- 55/100