ec2 ocf resource retry

Concerns: cluster-glue/lib/plugins/stonith/external/ec2

Seems to me that there are no retry mechanism in the EC2 OCF script.
AWS EC2 API calls can be throttle if more than 10000 API request a seconds are made.
In this case the script would not report any status and consider the resource in a bad status ending up with the STONITH device getting stopped.

Performing a "resource cleanup" operation starts the STONITH again in operational state after such failures.

/var/log/messages
2021-09-16T16:02:04.751248+00:00 external/ec2(res_AWS_STONITH)[31700]: info: status check for is
<-- Missing instance status report after "is" keyword

2021-09-16T16:02:04.760725+00:00 external/ec2(res_AWS_STONITH)[31694]: WARN: Already fenced (Instance status = ). Aborting fence attempt.
2021-09-16T16:02:13.742017+00:00 external/ec2(res_AWS_STONITH)[32004]: ERROR: Operation status failed: 1

Maybe some kind of fault tolerance would be nice to have I guess.

Start by inspecting cluster-glue/lib/plugins/stonith/external/ec2 and the logged status-check path. Review how EC2 API failures are reported during throttling, then verify that transient failures are retried or handled without incorrectly stopping the STONITH device. The log examples in the issue provide the failure behavior to reproduce or test against.

Written by the indexing model from the issue text.