Investigate why clusterctl sometimes does not move CAPV credentials secret
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 328
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 9
Description
Starting v1beta1, `VSphereCluster` API has a field [`IdentityRef` that points to the Secret containing vsphere creds](https://github.com/aws/eks-anywhere/blob/main/pkg/providers/vsphere/config/template-cp.yaml#L38).
This Secret should get moved from bootstrap to the workload cluster and vice versa depending on which operation (create/upgrade/delete) is being performed.
It is linked to the Cluster object through an ownerRef chain as follows:
* Once this Secret is created, the [capv controller updates it and adds the VSphereCluster object as an ownerRef](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/master/controllers/vspherecluster_reconciler.go#L245) on it. (Secret.OwnerReferences -> VSphereCluster)
* The VSphereCluster object itself gets the ownerRef of the Cluster object. (VSphereCluster.OwnerReferences -> Cluster)
Clusterctl move only moves those Secrets[ that meet certain conditions](https://cluster-api.sigs.k8s.io/clusterctl/provider-contract.html#move), one of them being that the Secret is linked to the Cluster object. Even though the CAPV credentials secret is linked through the ownerReferences chain described above to the Cluster object, in some end to end tests it did not get moved from the workload cluster to the bootstrap cluster during delete. This left the VSphereCluster in a not-ready state because it could not connect to the VCenter due to creds not being available on the cluster.
* Logs from one of the tests indicated that the update call for adding the OwnerRef on the Secret might have failed [here](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/master/controllers/vspherecluster_reconciler.go#L275). And the Secret was left on the workload cluster instead of being moved to the boostrap cluster and it had no OwnerRef.
* As a workaround we explicitly add the [clusterctl move label](https://github.com/aws/eks-anywhere/blob/main/pkg/providers/vsphere/config/template-cp.yaml#L531) to this Secret for now.
To do:
- [ ] Investigate why the OwnerRef did not get added, if the controller update failed and does not get retried
- [ ] Remove the clusterctl move label from template-cp.yaml
Contributor guide
Research direction
Start with the CAPV VSphereCluster reconciler update call linked in the issue and the owner-reference chain described for the credentials Secret. Review the workaround label in pkg/providers/vsphere/config/template-cp.yaml and the cited end-to-end test logs to determine whether a failed update is retried. Done means explaining the missing OwnerRef and removing the workaround label without breaking clusterctl move during delete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100