awslabs / awslabs/aws-lambda-redshift-loader

Better error logging in batches

Open
#50 1 comment 0 reactions 1 assignee Claimed by @IanMeyers View on GitHub
Dominant language
JavaScript
Stars
595
Forks
161
PR merge metrics
No merged PRs in 30d

Description

This is more a feature request than issue report:

_As operations monitoring the Lambda-based ETL process, I'd like the failed batches that are stored in DynamoDb to include more diagnostic information._

We had some loading errors in a newly set up table, and the information available in the batches are just:

```
{
"xxxx.redshift.amazonaws.com": {
"error": {
"code": "0A000",
"file": "/home/awsrsqa/padb/src/pg/src/backend/commands/commands_copy.c",
"length": 138,
"line": "2168",
"name": "error",
"routine": "DoCopy",
"severity": "ERROR"
},
"status": -1
}
}
```

We did manage to eventually find the corresponding log stream and diagnose it further, which was caused by a wrong copy command (which I suspect is yet another issue on its own), and quite a lot of more detailed error logs were actually available.

So here I wonder, would it be possible to include more details in the failed batches, so we can easily query it with just 'queryBatches.js'?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.