Specify Dask cluster to use conda environment upon connecting to it via Client

Hello,

We have a set up where we have a node (AWS SageMaker Notebook instance) that we use to coordinate various ML tasks. One task consists of using Dask for some at-scale data preprocessing. For this task, in a Jupyter notebook cell we would do...

`
client = Client(scheduler-ip)
`

The scheduler-ip is pointing to an already-created cluster (for example ECS). The cluster consists of nodes running Dask-scheduler and Dask-worker processes.

Now, the user using the Notebook knows the set of libraries that they want Dask cluster to understand / use. So the user would use conda-pack to package up those libraries; e.g. create `my_env.tar.gz` file.

What the user would like to then do is tell the Dask cluster to use that packaged-up environment.

Something like..

`
client.use_environment('my_env.tar.gz')
`

Is this possible to achieve?

I saw client.run / client.upload_file APIs. So I thought one potential would be to upload the tar.gz file then call "run" for Dask workers to use the environment or something. But that doesn't seem like a good solution in case of worker restarts / new workers.

Any pointers would be great.

Thank you!