Start a Dask cluster with EMR cluster programmatically
Open
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
We want to programmatically spin up an EMR cluster then spin up a Dask cluster in the EMR cluster with YarnCluster construct.
Currently, what we are doing is - open up SSH tunnel to the master node of the EMR cluster (it's in private subnet), log onto the master node, create a .ipynb notebook that has "YarnCluster(..)" code. We execute that cell to spin up the Dask cluster.
It would be nice to automate this; e.g. run some commands to spin up an EMR cluster that also has Dask cluster.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.