kubernetes / kubernetes/cloud-provider-openstack

[occm] Change to clusterrole introduced in v2.36.2 prevents deletion of non-existing nodes

Open Beginner friendly
#3,225 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug
Dominant language
Go
Stars
720
Forks
687
Avg merge
1d 14h
Merged PRs (30d)
51

Description

/kind bug

What happened:
After upgrading to occm chart >= v2.36.2 I am facing issues during rolling cluster upgrades that orphaned nodes (already deleted openstack vms) remain indefinitely in the cluster because occm does not have the permissions to remove the node(s) from the cluster.

The change was introduced here by @stephenfin.

What you expected to happen:
I would expect that the cloud-controller-manager has the permissions to delete unavailable / deleted nodes from the cluster.

That is also suggested here https://kubernetes.io/docs/concepts/architecture/cloud-controller/#authorization-miscellaneous but maybe a bit outdated.

How to reproduce it:
I am using cluster-api to manage my kubeadm clusters on openstack. To reproduce it trigger a rolling cluster upgrade e.g. bump k8s version. Then observe:

k get events -A --sort-by '{.metadata.creationTimestamp}' | grep DeletingNodeFailed
default                32m         Warning   DeletingNodeFailed             node/saas01-dev-nsc02-cp-cnhbb                                Failed deleting node saas01-dev-nsc02-cp-cnhbb: nodes "saas01-dev-nsc02-cp-cnhbb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
default                27m         Warning   DeletingNodeFailed             node/saas01-dev-nsc02-cp-cnhbb                                Failed deleting node saas01-dev-nsc02-cp-cnhbb: nodes "saas01-dev-nsc02-cp-cnhbb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
default                61s         Warning   DeletingNodeFailed             node/saas01-dev-nsc02-cp-cnhbb                                Failed deleting node saas01-dev-nsc02-cp-cnhbb: nodes "saas01-dev-nsc02-cp-cnhbb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
default                15s         Warning   DeletingNodeFailed             node/saas01-dev-nsc02-cp-qzgqn                                Failed deleting node saas01-dev-nsc02-cp-qzgqn: nodes "saas01-dev-nsc02-cp-qzgqn" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot delete resource "nodes" in API group "" at the cluster scope
k get nodes -o wide
NAME                                  STATUS                        ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                             KERNEL-VERSION     CONTAINER-RUNTIME
saas01-dev-nsc02-cp-cnhbb             NotReady,SchedulingDisabled   control-plane   29d   v1.35.7   10.42.0.113   <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-cp-njgkg             Ready                         control-plane   30m   v1.35.8   10.42.0.73    <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-cp-prjbj             Ready                         control-plane   27m   v1.35.8   10.42.0.185   <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-cp-qzgqn             NotReady,SchedulingDisabled   control-plane   29d   v1.35.7   10.42.0.214   <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-cp-zh6xw             Ready                         control-plane   36m   v1.35.8   10.42.0.171   <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-w-nbg1-fm9pp-ft4fx   Ready                         <none>          24m   v1.35.8   10.42.0.45    <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-w-nbg3-vgczw-7k2vz   Ready                         <none>          20m   v1.35.8   10.42.0.129   <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5
saas01-dev-nsc02-w-nbg6-69zsh-54fp5   Ready                         <none>          15m   v1.35.8   10.42.0.150   <none>        Flatcar Container Linux by Kinvolk 4593.2.5 (Oklo)   6.12.102-flatcar   containerd://2.1.5

This can only be resolved by manual intervention (kubectl delete node ...).

Anything else we need to know?:
Normally cluster-api will delete the nodes as soon a replacement node is ready. This mostly happens during control-plane upgrades when the API may transiently is not available.

Environment:

  • openstack-cloud-controller-manager(or other related binary) version: chart version v2.36.5
  • OpenStack version: 2026.1
  • Others: cluster-api v1.13.6; cluster-api-provider-openstack v0.14.8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked commit and the occm chart's ClusterRole for the cloud-controller-manager service account. Reproduce the rolling upgrade and inspect the DeletingNodeFailed events; done means the controller can delete unavailable nodes without the forbidden-resource error and orphaned nodes are removed automatically.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.