dask / dask/dask-blog

Blogpost idea: how to choose good settings for Dask on HPC

Open
#118 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
HTML
Stars
30
Forks
35
PR merge metrics
No merged PRs in 30d

Description

It'd be good to have a blogpost about how to choose good settings for Dask on HPC. Users are often confused about this.

I think one reason this is particularly confusing is that settings often need to be defined in multiple locations, and people are confused about how they interact. For example, someone might submit a job to SLURM with sbatch, which then runs a python program involving Dask, and want to know how that fits together.

https://github.com/dask/dask-blog/issues/116#issuecomment-947370655

> ...you know what would ALSO be a good blogpost? How to choose good cluster settings. Eg: how your SLURM/PBS/whatever batch submission settings relate to the settings you need to put in your dask-jobqueue cluster object.
>
> To be honest I'm still a bit confused by this, and it is something other people ask me too.
>
> If either @jacobtomlinson or @ian-r-rose would like to help make this, that would be very useful to refer people to (hint, hint) :smile:

@guillaumeeb has kindly agreed to help put this together https://github.com/dask/dask-blog/issues/116#issuecomment-947955079
> Hi all, I saw this issue, and I agree that both ideas would make great articles. Those are questions we see a lot as HPC admin/experts.
>
>I can try to help with the second one one batch submission settings! Everyone is confused about it.

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked discussion in dask-blog issue 116 and the examples here involving SLURM, PBS, sbatch, Python, and Dask job-queue settings. The goal is a blog post explaining how batch-submission settings and Dask cluster settings interact, but the issue names no source files, tests, or publication entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, hpc
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.