Azure / Azure/cyclecloud-slurm

Autoscaling doesn't work for pending jobs

Open
#11 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
84
Forks
56
Avg merge
3d 17h
Merged PRs (30d)
1

Description

I am using cyclecloud with a SLURM queueing system in combination with `dask` (a Python package that manages the bookkeeping and task distribution such that one doesn't have to write jobscripts and collect the data.)

It's possible to use that adaptive scaling feature of `dask` which means, that as the load on all my nodes becomes high, it automatically creates new jobs using (in my case) the following job script:
```
#!/bin/bash

#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -e TMPDIR/dask-worker-%J.err
#SBATCH -o TMPDIR/dask-worker-%J.out
#SBATCH -p debug
#SBATCH -A WAL
#SBATCH -n 1
#SBATCH --cpus-per-task=1
#SBATCH --mem=15G
#SBATCH -t 240:00:00
JOB_ID=${SLURM_JOB_ID%;*}

export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1

/gscratch/home/t-banij/miniconda3/envs/py37_min/bin/python -m distributed.cli.dask_worker tcp://10.75.64.5:43267 --nthreads 1 --memory-limit 16.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60 --local-directory TMPDIR
```

This results in many `PENDING` jobs. Unfortunately, no new nodes are started automatically, I still need to go into the web interface and start new nodes by hand, rendering the autoscale feature I am using useless.

Seems related to #1, #5, and #9.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the setup using the provided SLURM job script with Dask adaptive scaling, then review related issues #1, #5, and #9. Check why PENDING jobs do not cause new nodes to start through the web interface; done means queued jobs trigger node startup automatically without manual intervention.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.