Submit (long running) Yarn Cluster via CLI w/out Client Script
- Dominant language
- Python
- Stars
- 69
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Is there a way to use the CLI tools to submit a new YARN application and cluster, without creating a dask client script?
I know I can just add a while loop to the client script to run indefinitely, but it seems there should be a way to do something like:
```
dask-yarn start_cluster
```
I tried to just use `dask-yarn services scheduler/worker` but it complains about now being inside a YARN container:
```
Traceback (most recent call last):
File "/home/hadoop/miniconda/bin/dask-yarn", line 11, in
sys.exit(main())
File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/cli.py", line 412, in main
func(**kwargs)
File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/cli.py", line 289, in scheduler
app_client = skein.ApplicationClient.from_current()
File "/home/hadoop/miniconda/lib/python3.6/site-packages/skein/core.py", line 1046, in from_current
raise context.ValueError("Not running inside a container")
ValueError: Not running inside a container
```
Contributor guide
Assessment
This issue has not been assessed yet.