awslabs / awslabs/aws-lambda-redshift-loader
Additional SQS retry strategy for failed batches
- Dominant language
- JavaScript
- Stars
- 595
- Forks
- 161
- PR merge metrics
- No merged PRs in 30d
Description
The failedBatchReprocessingLambda provides for the option of subscribing to an SNS topic to reprocess failed batches.
I would like to defer attempts to re-process failed batches, for example during Redshift Maintenance Windows where immediately re-attempting to re-process batches will not make sense. There are SNS delivery policies such as minimum delay that could achieve this aim. The challenge is that I would like a subscriber that receives the failure event without delay but re-process the batch with a delay.
This could be achieved by subscribing an SQS queue to the subscriber and enhancing the failedBatchReprocessingLambda to be able to process SQS OR SNS events. In this way I could have one subscriber that is notified without delay and then set a "Delivery Delay" on an SQS queue. I would like to subscribe the failedBatchReprocessingLambda to the SQS queue that contains the original SNS failure message.
Contributor guide
Assessment
This issue has not been assessed yet.