awslabs / awslabs/aws-lambda-redshift-loader

Additional SQS retry strategy for failed batches

Open
#185 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
595
Forks
161
PR merge metrics
No merged PRs in 30d

Description

The failedBatchReprocessingLambda provides for the option of subscribing to an SNS topic to reprocess failed batches.

I would like to defer attempts to re-process failed batches, for example during Redshift Maintenance Windows where immediately re-attempting to re-process batches will not make sense. There are SNS delivery policies such as minimum delay that could achieve this aim. The challenge is that I would like a subscriber that receives the failure event without delay but re-process the batch with a delay.

This could be achieved by subscribing an SQS queue to the subscriber and enhancing the failedBatchReprocessingLambda to be able to process SQS OR SNS events. In this way I could have one subscriber that is notified without delay and then set a "Delivery Delay" on an SQS queue. I would like to subscribe the failedBatchReprocessingLambda to the SQS queue that contains the original SNS failure message.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.