rbd-nbd: occasionally file system entering into read-only on nodeplugin restart

Requires below PR from ceph:

- [ ] https://github.com/ceph/ceph/pull/42609
```

STEP: restart rbd-nbd process on nodeplugin and continue IO after nodeplugin restart
Jun 23 15:02:27.261: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.63:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-aec55366 --namespace=cephcsi-e2e-aec55366 delete -f -'
Jun 23 15:02:27.387: INFO: stderr: "warning: deleting cluster-scoped resources, not scoped to the provided namespace\n"
Jun 23 15:02:27.387: INFO: stdout: "storageclass.storage.k8s.io \"csi-rbd-sc\" deleted\n"
Jun 23 15:02:27.394: INFO: ExecWithOptions {Command:[/bin/sh -c ceph fsid] Namespace:rook-ceph PodName:rook-ceph-tools-7467d8bf8-vjzwh ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:02:27.394: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:02:29.405: INFO: Waiting up to &PersistentVolumeClaim{ObjectMeta:{rbd-pvc rbd-3274 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*csi-rbd-sc,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} to be in Bound state
Jun 23 15:02:29.406: INFO: waiting for PVC rbd-pvc (0 seconds elapsed)
Jun 23 15:02:31.409: INFO: waiting for PVC rbd-pvc (2 seconds elapsed)
Jun 23 15:02:31.415: INFO: Waiting for PV pvc-42f62cbf-9e2f-4f7a-99e1-98768ebfb561 to bind to PVC rbd-pvc
Jun 23 15:02:31.415: INFO: Waiting up to 3m0s for PersistentVolumeClaims [rbd-pvc] to have phase Bound
Jun 23 15:02:31.418: INFO: PersistentVolumeClaim rbd-pvc found and phase=Bound (2.553576ms)
Jun 23 15:02:31.418: INFO: Waiting up to 3m0s for PersistentVolume pvc-42f62cbf-9e2f-4f7a-99e1-98768ebfb561 to have phase Bound
Jun 23 15:02:31.421: INFO: PersistentVolume pvc-42f62cbf-9e2f-4f7a-99e1-98768ebfb561 found and phase=Bound (2.800294ms)
Jun 23 15:02:31.434: INFO: Waiting up to csi-rbd-demo-pod to be in Running state
Jun 23 15:02:31.451: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (0 seconds elapsed)
Jun 23 15:02:33.460: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (2 seconds elapsed)
Jun 23 15:02:35.455: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (4 seconds elapsed)
Jun 23 15:02:37.456: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (6 seconds elapsed)
Jun 23 15:02:39.455: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (8 seconds elapsed)
Jun 23 15:02:41.456: INFO: csi-rbd-demo-pod app is in Pending phase expected to be in Running state (10 seconds elapsed)
Jun 23 15:02:43.462: INFO: ExecWithOptions {Command:[/bin/sh -c rbd ls --format=json --pool=replicapool] Namespace:rook-ceph PodName:rook-ceph-tools-7467d8bf8-vjzwh ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:02:43.462: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:02:44.297: INFO: LabelSelector for csi-rbdplugin daemonsets in namespace cephcsi-e2e-aec55366: app=csi-rbdplugin
Jun 23 15:02:44.301: INFO: ExecWithOptions {Command:[/bin/sh -c uname -a] Namespace:cephcsi-e2e-aec55366 PodName:csi-rbdplugin-bvszc ContainerName:csi-rbdplugin Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:02:44.301: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:02:44.497: INFO: uname -a: Linux minikube 4.19.182 #1 SMP Wed Jun 9 00:54:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Jun 23 15:02:44.504: INFO: ExecWithOptions {Command:[/bin/sh -c rpm -qa | grep rbd-nbd] Namespace:cephcsi-e2e-aec55366 PodName:csi-rbdplugin-bvszc ContainerName:csi-rbdplugin Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:02:44.504: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:02:45.245: INFO: rbd-nbd package version: rbd-nbd-16.2.4-0.el8.x86_64

Jun 23 15:02:45.250: INFO: ExecWithOptions {Command:[/bin/sh -c ps -eo 'cmd' | grep [r]bd-nbd] Namespace:cephcsi-e2e-aec55366 PodName:csi-rbdplugin-bvszc ContainerName:csi-rbdplugin Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:02:45.250: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:02:45.485: INFO: map command running before restart, mapCmd: /usr/bin/rbd-nbd --id cephcsi-rbd-node -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 --keyfile=/tmp/csi/keys/keyfile-113061691 map replicapool/csi-vol-a5bccd9c-d42b-11eb-991e-4664e1c21ce8 --try-netlink --reattach-timeout=180

Jun 23 15:02:45.495: INFO: ExecWithOptions {Command:[/bin/sh -c ceph auth get-key client.cephcsi-rbd-node] Namespace:rook-ceph PodName:rook-ceph-tools-7467d8bf8-vjzwh ContainerName:rook-ceph-tools Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:02:45.495: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:02:47.629: INFO: Running '/usr/bin/kubectl --server=https://192.168.39.63:8443 --kubeconfig=/root/.kube/config --namespace=cephcsi-e2e-aec55366 delete po -l app=csi-rbdplugin --ignore-not-found=false'
Jun 23 15:02:57.864: INFO: stderr: ""
Jun 23 15:02:57.864: INFO: stdout: "pod \"csi-rbdplugin-bvszc\" deleted\n"
Jun 23 15:02:57.864: INFO: Waiting up to 10m0s for all daemonsets in namespace 'cephcsi-e2e-aec55366' to start
Jun 23 15:02:57.884: INFO: 0 / 1 pods ready in namespace 'cephcsi-e2e-aec55366' in daemonset 'csi-rbdplugin' (0 seconds elapsed)
Jun 23 15:02:59.888: INFO: 0 / 1 pods ready in namespace 'cephcsi-e2e-aec55366' in daemonset 'csi-rbdplugin' (2 seconds elapsed)
Jun 23 15:03:01.889: INFO: 1 / 1 pods ready in namespace 'cephcsi-e2e-aec55366' in daemonset 'csi-rbdplugin' (4 seconds elapsed)
Jun 23 15:03:01.889: INFO: attach command to run after restart, attachCmd: /usr/bin/rbd-nbd --id cephcsi-rbd-node -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 --keyfile=/tmp/csi/keys/keyfile-test attach --device /dev/nbd0 replicapool/csi-vol-a5bccd9c-d42b-11eb-991e-4664e1c21ce8 --try-netlink --reattach-timeout=180

Jun 23 15:03:01.893: INFO: ExecWithOptions {Command:[/bin/sh -c echo AQD+PdNgc4hbCRAAIquzYZxtb8XiMs9fGYO/BA== > /tmp/csi/keys/keyfile-test] Namespace:cephcsi-e2e-aec55366 PodName:csi-rbdplugin-8kqnl ContainerName:csi-rbdplugin Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:03:01.893: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:03:02.072: INFO: ExecWithOptions {Command:[/bin/sh -c /usr/bin/rbd-nbd --id cephcsi-rbd-node -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 --keyfile=/tmp/csi/keys/keyfile-test attach --device /dev/nbd0 replicapool/csi-vol-a5bccd9c-d42b-11eb-991e-4664e1c21ce8 --try-netlink --reattach-timeout=180
] Namespace:cephcsi-e2e-aec55366 PodName:csi-rbdplugin-8kqnl ContainerName:csi-rbdplugin Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:03:02.072: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:03:02.426: INFO: ExecWithOptions {Command:[/bin/sh -c ps -eo 'cmd' | grep [r]bd-nbd] Namespace:cephcsi-e2e-aec55366 PodName:csi-rbdplugin-8kqnl ContainerName:csi-rbdplugin Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:03:02.426: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:03:02.638: INFO: attach command running after restart, runningAttachCmd: /usr/bin/rbd-nbd --id cephcsi-rbd-node -m rook-ceph-mon-a.rook-ceph.svc.cluster.local:6789 --keyfile=/tmp/csi/keys/keyfile-test attach --device /dev/nbd0 replicapool/csi-vol-a5bccd9c-d42b-11eb-991e-4664e1c21ce8 --try-netlink --reattach-timeout=180

Jun 23 15:03:02.643: INFO: ExecWithOptions {Command:[/bin/sh -c echo 'Hello World' > /var/lib/www/html/test] Namespace:rbd-3274 PodName:csi-rbd-demo-pod ContainerName:web-server Stdin: CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false}
Jun 23 15:03:02.643: INFO: >>> kubeConfig: /root/.kube/config
Jun 23 15:03:18.124: INFO: stdErr occurred: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system

Jun 23 15:03:18.125: FAIL: failed to write IO, err: command terminated with exit code 2, stdErr: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system
```
failed at https://github.com/ceph/ceph-csi/pull/2095
Logs at https://jenkins-ceph-csi.apps.ocp.ci.centos.org/blue/rest/organizations/jenkins/pipelines/mini-e2e_k8s-1.20/runs/1270/nodes/87/steps/90/log/?start=0