Azure / Azure/cyclecloud-slurm
Autoscaling doesn't work for pending jobs
- Dominant language
- Python
- Stars
- 84
- Forks
- 56
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 1
Description
I am using cyclecloud with a SLURM queueing system in combination with `dask` (a Python package that manages the bookkeeping and task distribution such that one doesn't have to write jobscripts and collect the data.)
It's possible to use that adaptive scaling feature of `dask` which means, that as the load on all my nodes becomes high, it automatically creates new jobs using (in my case) the following job script:
```
#!/bin/bash
#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -e TMPDIR/dask-worker-%J.err
#SBATCH -o TMPDIR/dask-worker-%J.out
#SBATCH -p debug
#SBATCH -A WAL
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=15G
#SBATCH -t 240:00:00
JOB_ID=${SLURM_JOB_ID%;*}
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1
/gscratch/home/t-banij/miniconda3/envs/py37_min/bin/python -m distributed.cli.dask_worker tcp://10.75.64.5:43267 --nthreads 1 --memory-limit 16.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60 --local-directory TMPDIR
```
This results in many `PENDING` jobs. Unfortunately, no new nodes are started automatically, I still need to go into the web interface and start new nodes by hand, rendering the autoscale feature I am using useless.
Seems related to #1, #5, and #9.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the setup using the provided SLURM job script with Dask adaptive scaling, then review related issues #1, #5, and #9. Check why PENDING jobs do not cause new nodes to start through the web interface; done means queued jobs trigger node startup automatically without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100