dask / dask/helm-chart

Feature request: Allow separately specifying --nthreads and resource limits

Open
#63 1 comment 0 reactions 0 assignees View on GitHub
chart/dask
Dominant language
YAML
Stars
100
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I'm using Dask to coordinate some ML jobs, which have internal parallelism. I'd like to restrict my cluster so that only one job can run on a node at a time, even though nodes have multiple CPUs. However, this is currently impossible, since the Kubernetes worker resource limit is used to customize `--nthreads`:
https://github.com/dask/helm-chart/blob/master/dask/templates/dask-worker-deployment.yaml#L37-L38

This also prevents giving nodes e.g. 0.5 or 1.5 cpus, because even though Kubernetes allows it Dask will get confused and crash (i.e. `--nthreads` fails to parse as an integer.)

This is moderately esoteric so if I should just fork the chart and use that I don't mind.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.