dask / dask/dask-yarn

Start a Dask cluster with EMR cluster programmatically

Open
#121 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Hello,

We want to programmatically spin up an EMR cluster then spin up a Dask cluster in the EMR cluster with YarnCluster construct.

Currently, what we are doing is - open up SSH tunnel to the master node of the EMR cluster (it's in private subnet), log onto the master node, create a .ipynb notebook that has "YarnCluster(..)" code. We execute that cell to spin up the Dask cluster.

It would be nice to automate this; e.g. run some commands to spin up an EMR cluster that also has Dask cluster.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.