dask / dask/dask-yarn

Conda environment does not activate

Open
#134 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Hi,

The [current method](https://yarn.dask.org/en/latest/environments.html#using-python-environments-local-to-each-node) for starting a conda environment local to each node does not work correctly:

```python
from dask_yarn import YarnCluster

# Use a conda environment at /path/to/my/conda/env
cluster = YarnCluster(environment='conda:///path/to/my/conda/env')
```

It seems the environment does not activate, as discussed [here](https://github.com/conda/conda/issues/7980).

I am running dask-yarn 0.8.1.

A possible workaround (I have been using successfully) is to replace line 126 in dask_yarn/core.py:
```python
setup = "conda activate %s" % path
```
with the following:
```python
conda_root, conda_env = path.split("/envs/")
setup = "source %s/etc/profile.d/conda.sh && conda activate %s" % (conda_root, conda_env)
```
for environments stored in a standard location such as `/opt/anaconda3/envs/my_conda_env`.

The cluster can then be then started with:
```python
cluster = YarnCluster(environment='conda:///opt/anaconda3/envs/my_conda_env')
```

Please let me know if you would like me to submit a PR with the above.

Cheers, Roger

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.