dask / dask/dask-cloudprovider

Add documentation for setting environment variables (e.g. for installing extra packages)

Open
#169 7 comments 0 reactions 0 assignees View on GitHub
enhancement needs documentation provider/gcp/vm
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

I could be wrong, but this seems like a fairly important detail in some deployments. [Creating custom OS images with Packer](https://cloudprovider.dask.org/en/latest/packer.html) is obviously better but that's a fairly high bar for a lot of users, probably. I found that this is currently supported with the following caveats:

1. It cannot be configured in `cloudprovider.yaml` or environment variables on the client (i.e. `DASK_CLOUDPROVIDER__GCP__ENV_VARS` would be ignored)
2. If passed directly in code, it must be quoted. For example, ```GCPCluster(..., env_vars=dict(EXTRA_CONDA_PACKAGES="numba xarray"))``` will not work because the render code ([here](https://github.com/dask/dask-cloudprovider/blob/master/dask_cloudprovider/generic/cloud-init.yaml.j2#L46)) does not quote values and since the docker run statement built is already being yaml-escaped via single quotes, the values must be double quoted. The variables must be passed like this: ```EXTRA_CONDA_PACKAGES="\"numba xarray\""```, not ```EXTRA_CONDA_PACKAGES="'numba xarray'"```.

Documentation for new users on this would be very helpful. Allowing the `env_vars` to come from `cloudprovider.yaml` would also be nice.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.