dask / dask/dask-yarn

Submit (long running) Yarn Cluster via CLI w/out Client Script

Open
#90 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
69
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Is there a way to use the CLI tools to submit a new YARN application and cluster, without creating a dask client script?

I know I can just add a while loop to the client script to run indefinitely, but it seems there should be a way to do something like:
```
dask-yarn start_cluster
```

I tried to just use `dask-yarn services scheduler/worker` but it complains about now being inside a YARN container:

```
Traceback (most recent call last):
File "/home/hadoop/miniconda/bin/dask-yarn", line 11, in
sys.exit(main())
File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/cli.py", line 412, in main
func(**kwargs)
File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/cli.py", line 289, in scheduler
app_client = skein.ApplicationClient.from_current()
File "/home/hadoop/miniconda/lib/python3.6/site-packages/skein/core.py", line 1046, in from_current
raise context.ValueError("Not running inside a container")
ValueError: Not running inside a container
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.