Azure / Azure/local-csi-driver

cluster autoscaler not working with csistoragecapacities for node-local data

Open
#495 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
33
Forks
16
Avg merge
10h 17m
Merged PRs (30d)
28

Description

With CSIStorageCapacity enabled, CA's scale-up simulation runs the scheduler's VolumeBinding plugin against a template node for each nodegroup. During Filter, that plugin requires at least one CSIStorageCapacity whose nodeTopology LabelSelector matches the template's labels (this is the hasEnoughCapacity / nodeHasAccess check inside the plugin).

local-csi-driver's external-provisioner publishes one CSIStorageCapacity per node, keyed on topology.localdisk.csi.acstor.io/node=. This storage is local to the node, so that will help the csi driver places pods on nodes that still have available storage.

That value is tied to a specific node name, so when the scale-up simulation copies on of the existing nodes, it's also copying our csi-drivers node topology labels from an existing node. The template node will have the name of what ever node it copied in out csi driver's topology label.
This causes the VolumeBinding plugin to look at the available capacity for an existing node, which may be full or accidentally have enough space to fit the volume. This behavior is described a bit in this PR where there was an attempt to fix this issue:
https://github.com/kubernetes/autoscaler/pull/3887#issuecomment-909324444

As a result, the cluster autoscaler is not performing correctly right now with node-local csistoragecapacity

Contributor guide

Open the contributing guide

Research direction

Start with the issue's description of the Cluster Autoscaler scale-up simulation and the scheduler VolumeBinding plugin, including its hasEnoughCapacity and nodeHasAccess checks. Read the referenced autoscaler PR comment for prior context; done means scale-up simulation handles node-local CSIStorageCapacity without relying on a copied existing node's topology label.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.