Conda environment does not activate
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
The [current method](https://yarn.dask.org/en/latest/environments.html#using-python-environments-local-to-each-node) for starting a conda environment local to each node does not work correctly:
```python
from dask_yarn import YarnCluster
# Use a conda environment at /path/to/my/conda/env
cluster = YarnCluster(environment='conda:///path/to/my/conda/env')
```
It seems the environment does not activate, as discussed [here](https://github.com/conda/conda/issues/7980).
I am running dask-yarn 0.8.1.
A possible workaround (I have been using successfully) is to replace line 126 in dask_yarn/core.py:
```python
setup = "conda activate %s" % path
```
with the following:
```python
conda_root, conda_env = path.split("/envs/")
setup = "source %s/etc/profile.d/conda.sh && conda activate %s" % (conda_root, conda_env)
```
for environments stored in a standard location such as `/opt/anaconda3/envs/my_conda_env`.
The cluster can then be then started with:
```python
cluster = YarnCluster(environment='conda:///opt/anaconda3/envs/my_conda_env')
```
Please let me know if you would like me to submit a PR with the above.
Cheers, Roger
Contributor guide
Assessment
This issue has not been assessed yet.