How should we use pyjulia in parallel?
- Dominant language
- Python
- Stars
- 893
- Forks
- 101
- PR merge metrics
- No merged PRs in 30d
Description
I want to propose a discussion about how `pyjulia` should be used in parallel computing.
I recently found a way that seems rather stable:
1. we should make sure that code is loaded only once per each process
2. we should use `w+` mode for `memmapping`
Here is a mini snippet with a decorator that could be added to pyjulia to ensure point 1.
For point 2, instead, `joblib` has an option in `Parallel`. Should we may explicit this in the documentation?
```python
def julia_import(module: str, filename: str):
def decorator(fn):
def wrapper(*args, **kwargs):
# including stuffs
from julia import Main
if not hasattr(Main, module):
Main.include(filename)
return fn(Main, *args, **kwargs)
return wrapper
return decorator
@julia_import("MyModule", "lib.jl")
def python_function_using_julia(Main, *args, **kwargs):
return Main.MyModule.julia_function(*args, **kwargs)
res = Parallel(n_jobs=-1, mmap_mode="w+")(
delayed(python_function_using_julia)(arr) for arr in tqdm(data))
```
Example from: https://github.com/00sapo/pyjulia-vs-juliacall/blob/master/test.py
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.