dask / dask/dask-gateway

Helm deployment with multiple worker configuration profiles

Open
#470 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

In an effort to provide cost-optimization on the cluster scaling, I've been trying to schedule worker pods on to different sized nodes. I have two nodepools: small and large. I set the default pod limits to 1vcpu and 11GB RAM (wrt RAM, two fit on a small node and four would fit on to a large node), however, I also want some users to, through the `cluster.options` configuration when starting a cluster, to request higher limits (3vcpus and 21GB RAM for a full small node or even 7vcpu and 45GB RAM for a large node).

Below is the `NodeAffinity` solution that kind of works, however, when a large enough cluster is scaled up with the default limits (1vcpu and 11GB RAM), sometimes this triggers the scale up of the larger nodes. This isn't a surprise due to the `kube-scheduler` and the non-guarantee of the `preferredDuringSchedulingIgnoredDuringExecution` preferences.

```yaml
dask-gateway:
gateway:
baskend:
worker:
extraPodConfig:
tolerations:
- key: node_toleration
value: worker
operator: Equal
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node_label
operator: In
values:
- worker
- lgworker
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node_label
operator: In
values:
- worker
- weight: 1
preference:
matchExpressions:
- key: node_label
operator: In
values:
- lgworker

```

So, I was wondering, is it feasible to have multiple profiles for worker pods? Something that would effectively extend the cluster profile options [here](https://gateway.dask.org/cluster-options.html#cluster-profiles).

Otherwise, perhaps someone may have some better insight in how to achieve what I'm trying to accomplish.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.