JuliaPy / JuliaPy/PythonCall.jl
segmentation fault with multi-threading
- 主要语言
- Julia
- 星标
- 1.1k
- 派生
- 86
- 平均合并
- 1 天 22 小时
- 30 天内合并 PR
- 3
描述
I have this MWE, where I get segmentation faults (frequently, but not deterministically), when trying to run some script that uses multi-threading on the Julia side.
I have used before launching `ipython3`:
```bash
export JULIA_NUM_THREADS=4
```
(my computer has 4 cores - 8 threads).
The MWE is:
```python
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from juliacall import Main as jl
In [2]: import numpy as np
In [3]: jl.seval("""
...: function test(x)
...: partial = zeros(Threads.nthreads())
...: Threads.@threads for i in 1:Threads.nthreads()
...: for j in i:Threads.nthreads():length(x)
...: partial[i] += x[j]
...: end
...: end
...: return sum(partial)
...: end
...: """)
Out[3]: test (generic function with 1 method)
In [4]: x = np.random.random((10_000,))
In [5]: %timeit jl.test(x)
72.2 µs ± 35.9 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [6]: %timeit jl.test(x)
Segmentation fault (core dumped)
```
Here I have emulated the error using the `%timeit` macro from `ipython`, but my actual error I get after some runs of a function of my package:
```python
In [1]: from juliacall import Main as jl
In [2]: jl.seval("using CellListMap")
In [3]: import numpy as np
In [5]: x = np.random.random((10_000,3))
In [6]: nb = jl.neighborlist(x.transpose(), 0.05)
In [7]: nb = jl.neighborlist(x.transpose(), 0.05)
In [8]: nb = jl.neighborlist(x.transpose(), 0.05)
In [9]: nb = jl.neighborlist(x.transpose(), 0.05)
In [10]: nb = jl.neighborlist(x.transpose(), 0.05)
In [11]: nb = jl.neighborlist(x.transpose(), 0.05)
In [12]: nb = jl.neighborlist(x.transpose(), 0.05)
In [13]: nb = jl.neighborlist(x.transpose(), 0.05)
In [14]: nb = jl.neighborlist(x.transpose(), 0.05)
In [15]: nb = jl.neighborlist(x.transpose(), 0.05)
In [16]: nb = jl.neighborlist(x.transpose(), 0.05)
In [17]: nb = jl.neighborlist(x.transpose(), 0.05)
In [18]: nb = jl.neighborlist(x.transpose(), 0.05)
In [19]: nb = jl.neighborlist(x.transpose(), 0.05)
Segmentation fault (core dumped)
```
`%timeit` runs the function multiple times, there seems to be some memory corruption, or memory overflow, causing the error.
Anyway, even if you have only some hint on how to debug this, I will be very thankful.
(even in the simplest example above, the segfaults only occur with multi-threading).
贡献指南
这个仓库没有索引到贡献指南
调研方向
Start by reproducing the MWE from `from juliacall import Main as jl` with `JULIA_NUM_THREADS=4`, then compare repeated calls to `jl.test` under IPython `%timeit`. The issue is done when the threaded example and repeated `jl.neighborlist` calls no longer produce a segmentation fault, or the failure is isolated to a specific entry point.
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100