dask / dask/distributed

Change Python environment to distribute software

Open
#1,457 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Fun fact:** In Python 3 you can start Process objects from different Python installations

Example
--------

```python
In [1]: import sys

In [2]: sys.version
Out[2]: '3.6.0 |Anaconda custom (64-bit)| (default, Dec 23 2016, 12:22:00) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]'

In [3]: import foo

In [4]: foo.f??
Signature: foo.f()
Source:
def f():
print(sys.version)
File: ~/foo.py
Type: function

In [5]: import multiprocessing

In [6]: ctx = multiprocessing.get_context('forkserver')

In [7]: ctx.set_executable('/home/mrocklin/Software/anaconda/envs/35/bin/python')

In [8]: proc = ctx.Process(target=foo.f)

In [9]: proc.start()

3.5.3 | packaged by conda-forge | (default, Feb 9 2017, 14:37:12)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
```

How to use this
----------------

A long time ago we had a feature where you could distribute a relocatable conda environment to workers and then have those workers restart themselves in that environment. We removed this because it complicated the nannies. This method however may be much simpler. This need has come up twice recently:

1. In Yarn, where we create conda environments and ship them with Yarn
2. In dask-kubernetes, where we ask users to create their own docker containers

Adding some software environment management to dask would help to resolve both of these problems

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.