dask / dask/dask-cloudprovider
Plugins that help to pass credentials for S3 and GCS to remote cluster workers
Open
provider/aws/ec2
provider/gcp/vm
question
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
I didn't find a simple way to pass credentials to remote workers, such as S3 and GCS, while both are widely used to store data frames.
In this ticket's scope, I propose creating plugins that will help distribute the required keys to remote workers.
**GCP credentials**
GCP credentials file path is stored in `GOOGLE_APPLICATION_CREDENTIALS` env variable. The plugin has to create a remote file and pass an env variable with a proper path to workers.
**S3 credentials**
Like GCP, we must update credential files and store them on each worker.
PR: #439
Contributor guide
Assessment
This issue has not been assessed yet.