JuliaPy / JuliaPy/pyjulia

How should we use pyjulia in parallel?

Open
#472 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
893
Forks
101
PR merge metrics
No merged PRs in 30d

Description

I want to propose a discussion about how `pyjulia` should be used in parallel computing.

I recently found a way that seems rather stable:
1. we should make sure that code is loaded only once per each process
2. we should use `w+` mode for `memmapping`

Here is a mini snippet with a decorator that could be added to pyjulia to ensure point 1.
For point 2, instead, `joblib` has an option in `Parallel`. Should we may explicit this in the documentation?

```python
def julia_import(module: str, filename: str):
def decorator(fn):
def wrapper(*args, **kwargs):
# including stuffs
from julia import Main
if not hasattr(Main, module):
Main.include(filename)

return fn(Main, *args, **kwargs)
return wrapper
return decorator

@julia_import("MyModule", "lib.jl")
def python_function_using_julia(Main, *args, **kwargs):
return Main.MyModule.julia_function(*args, **kwargs)

res = Parallel(n_jobs=-1, mmap_mode="w+")(
delayed(python_function_using_julia)(arr) for arr in tqdm(data))
```

Example from: https://github.com/00sapo/pyjulia-vs-juliacall/blob/master/test.py

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.