awslabs / awslabs/aws-lambda-redshift-loader

Dataloss issue with this lambda redshift loader

Open
#230 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
595
Forks
161
PR merge metrics
No merged PRs in 30d

Description

Hi,
We have faced data loss issue with this setup and had to go with a different approach. Looking forward to get the issue fixed.

We have done all the set up and was working fine with relatively less data. We used the reprocessBatch approach for firing the copy command for failed files. It worked well in lower environment . But in production, with the large volume of data we faced dataloss of 850 million record out of 23 billion records. We had made sure that the failed files are reprocessed.
We didn't get any error also.

Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.