Azure / Azure/local-csi-driver
cluster autoscaler not working with csistoragecapacities for node-local data
- Dominant language
- Go
- Stars
- 33
- Forks
- 16
- Avg merge
- 10h 17m
- Merged PRs (30d)
- 28
Description
With CSIStorageCapacity enabled, CA's scale-up simulation runs the scheduler's VolumeBinding plugin against a template node for each nodegroup. During Filter, that plugin requires at least one CSIStorageCapacity whose nodeTopology LabelSelector matches the template's labels (this is the hasEnoughCapacity / nodeHasAccess check inside the plugin).
local-csi-driver's external-provisioner publishes one CSIStorageCapacity per node, keyed on topology.localdisk.csi.acstor.io/node=. This storage is local to the node, so that will help the csi driver places pods on nodes that still have available storage.
That value is tied to a specific node name, so when the scale-up simulation copies on of the existing nodes, it's also copying our csi-drivers node topology labels from an existing node. The template node will have the name of what ever node it copied in out csi driver's topology label.
This causes the VolumeBinding plugin to look at the available capacity for an existing node, which may be full or accidentally have enough space to fit the volume. This behavior is described a bit in this PR where there was an attempt to fix this issue:
https://github.com/kubernetes/autoscaler/pull/3887#issuecomment-909324444
As a result, the cluster autoscaler is not performing correctly right now with node-local csistoragecapacity
Contributor guide
Research direction
Start with the issue's description of the Cluster Autoscaler scale-up simulation and the scheduler VolumeBinding plugin, including its hasEnoughCapacity and nodeHasAccess checks. Read the referenced autoscaler PR comment for prior context; done means scale-up simulation handles node-local CSIStorageCapacity without relying on a copied existing node's topology label.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100