vSphere CSI volume attachment issues

**What happened**:
randomly pods cannot be created, I am getting a `volume is currently in use` error on container creation when restarting pods.

this issue is critical as the production cluster I am running is now becoming more and more unstable, I use renovatebot to keep deployment images and helm charts up to date, each day I am finding randomly pods that cannot start cause of this volume attachment issue, causing downtime, please make this a priority.

``
Warning FailedAttachVolume 17m attachdetach-controller AttachVolume.Attach failed for volume "pvc-e1610bdf-5420-444e-99bd-a274642b66e7" : rpc error: code = Internal desc = failed to attach disk: "6b284a9a-582d-4ecc-bb94-bf49017e587a" with node: "192.168.22.161" err failed to attach cns volume: "6b284a9a-582d-4ecc-bb94-bf49017e587a" to node vm: "VirtualMachine:vm-5190 [VirtualCenterHost: vcenter.vsphere.xxx.xxx UUID: 421d6be9-800f-38ca-557d-4fa14b7895a9, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: vcenter.vsphere.xxx.xxx]]". fault: "(*types.LocalizedMethodFault)(0xc000c4f8e0)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (*types.ResourceInUse)(0xc000c6d140)({\n VimFault: (types.VimFault) {\n MethodFault: (types.MethodFault) {\n FaultCause: (*types.LocalizedMethodFault)(),\n FaultMessage: ([]types.LocalizableMessage) \n }\n },\n Type: (string) \"\",\n Name: (string) (len=6) \"volume\"\n }),\n LocalizedMessage: (string) (len=32) \"The resource 'volume' is in use.\"\n})\n". opId: "101159e7"
``

**What you expected to happen**:
pod restarts to work without volume attachment issues

**How to reproduce it (as minimally and precisely as possible)**:
create a vSphere cluster, enable DRS, install deployment with a PVC that uses standard storage class, move eks worker nodes to different esxi nodes, restart pods

**Anything else we need to know?**:
everything was working fine till I enabled DRS in vCenter, after awhile this issue started to occur, I believe it has to do with DRS moving around the worker nodes and vSphere CSI not properly detaching/attaching CNS volumes

this is also similar to issue https://github.com/aws/eks-anywhere/issues/896

**Environment**:
- EKS Anywhere Release: 0.11.4
- EKS Distro Release: 1.23

Start by reproducing the failure with EKS Anywhere 0.11.4 and EKS Distro 1.23: enable vCenter DRS, use a deployment with a PVC on the standard storage class, move worker nodes between ESXi hosts, and restart pods. Compare the behavior with issue 896 and verify completion when pod restarts no longer produce volume attachment or resource-in-use errors after node movement.

Written by the indexing model from the issue text.