dask / dask/dask-cloudprovider
Add documentation for setting environment variables (e.g. for installing extra packages)
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
I could be wrong, but this seems like a fairly important detail in some deployments. [Creating custom OS images with Packer](https://cloudprovider.dask.org/en/latest/packer.html) is obviously better but that's a fairly high bar for a lot of users, probably. I found that this is currently supported with the following caveats:
1. It cannot be configured in `cloudprovider.yaml` or environment variables on the client (i.e. `DASK_CLOUDPROVIDER__GCP__ENV_VARS` would be ignored)
2. If passed directly in code, it must be quoted. For example, ```GCPCluster(..., env_vars=dict(EXTRA_CONDA_PACKAGES="numba xarray"))``` will not work because the render code ([here](https://github.com/dask/dask-cloudprovider/blob/master/dask_cloudprovider/generic/cloud-init.yaml.j2#L46)) does not quote values and since the docker run statement built is already being yaml-escaped via single quotes, the values must be double quoted. The variables must be passed like this: ```EXTRA_CONDA_PACKAGES="\"numba xarray\""```, not ```EXTRA_CONDA_PACKAGES="'numba xarray'"```.
Documentation for new users on this would be very helpful. Allowing the `env_vars` to come from `cloudprovider.yaml` would also be nice.
Contributor guide
Assessment
This issue has not been assessed yet.