dask / dask/distributed

Support for wheels when installing dependencies with PipInstall plugin

Open
#6,202 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Linked to #1200 and new PipInstall plugin in `distributed.diagnostic.plugin`.

Sometimes there is the need to deploy custom Python module to dask workers. Those uses cases are generally well covered by few methods:
- `client.upload_file` if we need to send a `.py`, `.egg` or `.zip` file
- `PipInstall` plugin if we need to install additional dependencies at workers startup
- various `Client.run` or custom plugins solutions for things like conda packages

However I may have found a case where none of those solutions are working.
In my case I developed few custom modules that are not zip_safe (data files or cython extensions)
I tried uploading an `egg`, `wheel` or `zip` file, but those did not work (getting `NotADirectoryError` for data files)

I also tried to pip install the wheel after uploading it, but that also does not work (silently)
After modifying the plugin by adding `logger = logging.getLogger("distributed.worker")` I got a __***.whl' looks like a filename, but the file does not exist__ error.

So, my question is: Is there a way to install a wheel using PipInstall? where are the file sent with `client.upload_file` stored on the workers?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.