One bad CSI volume can stop all volumes from being scheduled

While implementing CSI support for Hetzner Cloud we ran into some strange behaviour around cases where one "broken" volume can cause all other volumes to not schedule properly.

In the case of the Hetzner plugin this was caused by Volumes created in hel1 generally not being allowed to be scheduled in nbg1 - which is expected. But due to the fact that this volume was also tried to be published in the node in nbg1 (How would we prevent this? Do/Should Cluster volumes have support for placement constraints?) this will never work on these nodes. Another volume created in nbg1, which should succeed, was however blocked by the earlier failure from succeeding.

Only after force removing all volumes and services and starting from scratch (and this time making sure to only schedule stuff in nbg1), we were able to properly create a volume.

see https://github.com/hetznercloud/csi-driver/pull/376#issuecomment-1429775169 for further details

Start by tracing SwarmKit's volume scheduling and CSI publication handling, using the Hetzner CSI failure scenario described in the issue as the reproduction case. Check how a failed volume affects later eligible volumes, then add coverage showing that one unschedulable volume does not block another volume that can run; done means the independent nbg1 volume schedules successfully.