Limit number of jobs in queue or running with dask-jobqueue
Open
question
- Dominant language
- Python
- Stars
- 256
- Forks
- 150
- PR merge metrics
- No merged PRs in 30d
Description
Original problem raised by @jmichel-otb in https://github.com/dask/dask-jobqueue/issues/196#issuecomment-548412595.
> I currently encounter the limitation of maximum number of jobs in queue. Our maximum number of jobs in queue is 100, which is not much.
>
> I understand that using job arrays is off the table, but maybe we could make dask jobqueue aware of those limits, and inform it that it needs to start say 200 jobs but it needs to ensure that there are never more than 100 queuing ?
>
> I am not familiar with jobqueue code but I expect this to be a much smaller patch than enabling job arrays.
Contributor guide
Assessment
This issue has not been assessed yet.