dask / dask/distributed

Add CondaPipInstall WorkerPlugin

Open
#3,111 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Often users want to make small modifcations onto their existing environment. For large changes, we recommend affecting docker images, NFS installations, and so on, but for small changes we often find people doing something like the following:

```python
def install_foo():
os.system("pip install foo")

client.run(install_foo)
```

This can cause problems, particularly if workers restart, if new workers arise, or if the library is already around and imported into Python (and so the new version doesn't show up).

We should probably provide a more robust solution for this. I propose that we create a [`WorkerPlugin`](https://docs.dask.org/en/latest/setup/custom-startup.html#worker-lifecycle-plugins) that does this robustly. Something like the following:

```python
class CondaPipInstall:
def __init__(self, packages: List[Tuple[str, str]]):
self.packages = packages

def setup(self, worker):
if any of the packages aren't up to date:
install_packages(self.packages)
restart_this_worker()

install = CondaPipInstall(packages=["numpy >= 1.17", "git+https://github.com/...@master"])
client.register_worker_plugin(install)
```

I think that this would be a good middle ground between `client.run` or `client.upload_file` and asking users to make new docker images.

(this came up in conversation with @jcrist and @TomAugspurger)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.