MetaCell / MetaCell/cloud-harness

Implement high level node assignment control

Open
#479 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19
Forks
5
Avg merge
3d 18h
Merged PRs (30d)
3

Description

The main high level desire is to separate "service" pods by "on demand" and "user" pods.

One reason is we want to have a stable resources assignment to pods serving applications, removing the risk that greedy on demand pods coming from jupyterhub and workflows take over the resources.

Another reason is the possibility to exploit cheaper solutions for these on demand nodes, such as [GKE preemptible nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms).

Possible solutions are taints/tolerations and node affinity/antiaffinity.

JupyterHub is already configured to provide nodeSelectors and has the default affinity on `hub.jupyter.org/node-purpose=user`.
The natural implementation would be to create a default affinity for on-demand pods and an opposite anti-affinity for service pods.

So all jupyterhub pods and workflows will have the following affinity by default:

```yaml
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: cloudharness/node-purpose
operator: In
values:
- on-demand
weight: 100
```

While all deployments and statefulsets would have the antiaffinity:

```yaml
nodeAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: cloudharness/node-purpose
operator: In
values:
- on-demand
weight: 100
```

The implementation must set the above defaults, but also allow for more affinities and anti-affinities, including overriding the above for each deployment separately by configuring the values.yaml files.

In the main global helm/values.yaml file we configure the above behaviour with:

```yaml
affinities:
workflows:
- "cloudharness/node-purpose": "on-demand"
jupyterhub:
- "cloudharness/node-purpose": "on-demand"

```

While the default application's values.yaml will add the affinities/antiaffinities sections with the defaults:
```yaml
harness:
...
deployment:
affinities: []
antiAffinities:
- "cloudharness/node-purpose": "on-demand"
```

As we can have different Juptyerhub deployments, each one must be able to setup specific affinities and antiaffinities separately, in addition to the global values as specified above.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the main global helm/values.yaml and the default application's values.yaml, then inspect the Helm templates for JupyterHub, workflows, deployments, and statefulsets. Verify that rendered manifests apply the requested default affinities and anti-affinities while allowing per-application and per-JupyterHub overrides. Done means on-demand and service pods receive the intended scheduling preferences without preventing custom configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, yaml
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.