JuliaPy / JuliaPy/pyjulia

Crash when trying to use Julia from Dask multiprocessing

Open
#344 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
893
Forks
101
PR merge metrics
No merged PRs in 30d

Description

Hello,

PyJulia crashes when I try to callJulia from inside a Python process using Dask multiprocessing (dask.distributed.Client) throwing the following exceptions:
```
Exception: OSError('exception: access violation reading 0x00000000000019A8')
Exception: AttributeError("function 'was_initialized' not found")
```

Calling Julia from Python's built-in multiprocessing works however fine.

I am using Windows 10 (64 bit), Python 3.7, Dask 2.9 and Julia 1.3.

Test Code:

```
from julia import Julia
import dask.distributed
import concurrent.futures

def jl_calc(id):
func_text = """
function myfunc(n:: Int64)
z:: Int64=1
for i=1:n
z += 2i^2
end
z
end
"""
jl = Julia()
test_func = jl.eval(func_text)
res = test_func(id+1)
return res

def submit_tasks(executor):
futures = []
for i in range(10):
futures.append(executor.submit(jl_calc, i))
results = [future.result() for future in futures]
print(results)

if __name__ == '__main__':
print('calling Julia from Pythons bulit-in Multiprocessing')
executor_multiprocessing = concurrent.futures.ProcessPoolExecutor()
try:
submit_tasks(executor_multiprocessing)
finally:
executor_multiprocessing.shutdown()
print('calling Julia from Pythons bulit-in Multiprocessing successful')

print('calling Julia from Dask Client')
executor_dask = dask.distributed.Client()
try:
submit_tasks(executor_dask)
finally:
executor_dask.shutdown()
print('calling Julia from Dask Client successful')
```

System setup and stack trace is given in the attachment.
[Dask_Julia_error.txt](https://github.com/JuliaPy/pyjulia/files/3984013/Dask_Julia_error.txt)

Edit:

The same issue has been raised at Dask, because I do not know in which component the issue is:

https://github.com/dask/dask/issues/5732

Best Regards
Benjamin

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.