Support for wheels when installing dependencies with PipInstall plugin
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Linked to #1200 and new PipInstall plugin in `distributed.diagnostic.plugin`.
Sometimes there is the need to deploy custom Python module to dask workers. Those uses cases are generally well covered by few methods:
- `client.upload_file` if we need to send a `.py`, `.egg` or `.zip` file
- `PipInstall` plugin if we need to install additional dependencies at workers startup
- various `Client.run` or custom plugins solutions for things like conda packages
However I may have found a case where none of those solutions are working.
In my case I developed few custom modules that are not zip_safe (data files or cython extensions)
I tried uploading an `egg`, `wheel` or `zip` file, but those did not work (getting `NotADirectoryError` for data files)
I also tried to pip install the wheel after uploading it, but that also does not work (silently)
After modifying the plugin by adding `logger = logging.getLogger("distributed.worker")` I got a __***.whl' looks like a filename, but the file does not exist__ error.
So, my question is: Is there a way to install a wheel using PipInstall? where are the file sent with `client.upload_file` stored on the workers?
Contributor guide
Assessment
This issue has not been assessed yet.