Improve robustness during outages

Yesterday, the Hetzner Cloud API had an outage, and it appears that the docker machine driver did not handle it well.

You can see that from 2024-11-13 17:00:00 to 2024-11-14 08:00:00, the amount of requests to /server_types, /images and /locations is unexpectedly high. Also, the amount of requests for single action was also really high.

This leads into rate limits, while waiting for servers to be created.

I see a few possible improvements:

• When waiting for action, use an exponential back off algorithm to spread the requests over time. You can cap the max waiting time to a sensible value. https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#WithPollOpts https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#ExponentialBackoffWithOpts
• Use a single API call to wait for multiple related actions, using https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#ActionClient.WaitFor or https://pkg.go.dev/github.com/hetznercloud/hcloud-go/v2/hcloud#ActionClient.WaitForFunc (note that the Watch* API is deprecated).
• Maybe cache the call the /locations, /server types and /images, those shouldn't change that often. Unless you are checking for a server type availability ?

Start by tracing how the driver polls related actions and requests /server_types, /images, and /locations during server creation. Compare the current polling with the linked hcloud-go WaitFor and exponential-backoff options; done means outage waits spread requests over time without triggering rate limits.