kubernetes / kubernetes/cloud-provider-openstack
[occm] Change to clusterrole introduced in v2.36.2 prevents deletion of non-existing nodes
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 720
- Forks
- 687
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 51
Description
/kind bug
What happened:
After upgrading to occm chart >= v2.36.2 I am facing issues during rolling cluster upgrades that orphaned nodes (already deleted openstack vms) remain indefinitely in the cluster because occm does not have the permissions to remove the node(s) from the cluster.
The change was introduced here by @stephenfin.
What you expected to happen:
I would expect that the cloud-controller-manager has the permissions to delete unavailable / deleted nodes from the cluster.
That is also suggested here https://kubernetes.io/docs/concepts/architecture/cloud-controller/#authorization-miscellaneous but maybe a bit outdated.
How to reproduce it:
I am using cluster-api to manage my kubeadm clusters on openstack. To reproduce it trigger a rolling cluster upgrade e.g. bump k8s version. Then observe:
k get events -A --sort-by '{.metadata.creationTimestamp}' | grep DeletingNodeFailed
default 32m Warning DeletingNodeFailed node/saas01-dev-nsc02-cp-cnhbb Failed deleting node saas01-dev-nsc02-cp-cnhbb: nodes "saas01-dev-nsc02-cp-cnhbb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
default 27m Warning DeletingNodeFailed node/saas01-dev-nsc02-cp-cnhbb Failed deleting node saas01-dev-nsc02-cp-cnhbb: nodes "saas01-dev-nsc02-cp-cnhbb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
default 61s Warning DeletingNodeFailed node/saas01-dev-nsc02-cp-cnhbb Failed deleting node saas01-dev-nsc02-cp-cnhbb: nodes "saas01-dev-nsc02-cp-cnhbb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
default 15s Warning DeletingNodeFailed node/saas01-dev-nsc02-cp-qzgqn Failed deleting node saas01-dev-nsc02-cp-qzgqn: nodes "saas01-dev-nsc02-cp-qzgqn" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
saas01-dev-nsc02-cp-cnhbb NotReady,SchedulingDisabled control-plane 29d v1.35.7 10.42.0.113 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-cp-njgkg Ready control-plane 30m v1.35.8 10.42.0.73 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-cp-prjbj Ready control-plane 27m v1.35.8 10.42.0.185 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-cp-qzgqn NotReady,SchedulingDisabled control-plane 29d v1.35.7 10.42.0.214 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-cp-zh6xw Ready control-plane 36m v1.35.8 10.42.0.171 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-w-nbg1-fm9pp-ft4fx Ready <none> 24m v1.35.8 10.42.0.45 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-w-nbg3-vgczw-7k2vz Ready <none> 20m v1.35.8 10.42.0.129 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
saas01-dev-nsc02-w-nbg6-69zsh-54fp5 Ready <none> 15m v1.35.8 10.42.0.150 <none> Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo) 6.12.102-flatcar containerd://2.1.5
This can only be resolved by manual intervention (kubectl delete node ...).
Anything else we need to know?:
Normally cluster-api will delete the nodes as soon a replacement node is ready. This mostly happens during control-plane upgrades when the API may transiently is not available.
Environment:
- openstack-cloud-controller-manager(or other related binary) version: chart version v2.36.5
- OpenStack version: 2026.1
- Others: cluster-api v1.13.6; cluster-api-provider-openstack v0.14.8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked commit and the occm chart's ClusterRole for the cloud-controller-manager service account. Reproduce the rolling upgrade and inspect the DeletingNodeFailed events; done means the controller can delete unavailable nodes without the forbidden-resource error and orphaned nodes are removed automatically.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100