Better retry strategy for failed loads.

Hi Ian,

We are running into the situation where loads timeout at certain times of the day because the cluster is under heavy utilisation. Looking at #68 as reference, we’ve already tried to tune this as much as possible. Our lambda timeout is set to the max (5 minutes). Our BatchSize and BatchSizeBytes are set to arbitrarily high numbers (128, 128GB) such that our BatchTimeoutSecs (600) is what is triggering the loads.

Error logs show this is due to redshift timing out:
`Cluster Load Failure error: abort query on Cluster OurCluster.redshift.amazonaws.com`

We have tried to remedy this by manually running the copy queries and the reprocess batch script. This is time intensive and we are hoping for a way to automate this.

One suggestion we have is adding configuration options to the loader to automatically return failed files from a 'processed' state to a 'pending' state (not sure I'm using the right terminology), ready to be pick up again by the original loader. Some form of an automatic retry strategy. Something else good to have in this scenario is a configurable retry wait time, static or dynamic, to prevent loading into a maybe already overloaded cluster.

One other idea we had was to play around with the lambda's retry settings. Any thoughts on that?

Otherwise, do you have any other suggestion/ideas which might help?

Thanks!