dask / dask/dask-jobqueue

a direct way to specify the worker spec

Open
#601 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

I'm frequently confused by the way the API requires me to specify resources for the jobqueue: I need to specify the job size and the number of workers per job (there's quite a few more knobs, of course), and it would evenly distribute the resources to each worker. I can then choose how many jobs to submit.

However, as a user (with admittedly a limited knowledge of how HPC work, so what I'm describing may be naive), my view is usually something like:
- my local jobqueue allows individual jobs to request up to 115GB of memory, 28 cores and a certain walltime, and for some queues there's also minimum resource limits
- I want to have about 14 workers, with about 15 GB and 2 threads each, where the concrete worker specs are often somewhat arbitrary and depend on my knowledge of the problem I'm trying to compute

This usually leads to me trying to group the workers manually to optimally fit the resource limits (so I don't get de-prioritized by submitting too many jobs).

Instead, I ideally would like an API allows me to specify (or retrieve) the resource limits per job of the jobqueue and the desired worker spec. It would then try to optimally distribute the workers and submit the jobs for me (and fail early if the resource limits don't allow the worker spec I requested).

Does something like this exist already? If not, would you be open to adding something like that? Is there anything I'm missing that would inhibit something like this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.