awslabs / awslabs/aws-lambda-redshift-loader
Better error logging in batches
- Dominant language
- JavaScript
- Stars
- 595
- Forks
- 161
- PR merge metrics
- No merged PRs in 30d
Description
This is more a feature request than issue report:
_As operations monitoring the Lambda-based ETL process, I'd like the failed batches that are stored in DynamoDb to include more diagnostic information._
We had some loading errors in a newly set up table, and the information available in the batches are just:
```
{
"xxxx.redshift.amazonaws.com": {
"error": {
"code": "0A000",
"file": "/home/awsrsqa/padb/src/pg/src/backend/commands/commands_copy.c",
"length": 138,
"line": "2168",
"name": "error",
"routine": "DoCopy",
"severity": "ERROR"
},
"status": -1
}
}
```
We did manage to eventually find the corresponding log stream and diagnose it further, which was caused by a wrong copy command (which I suspect is yet another issue on its own), and quite a lot of more detailed error logs were actually available.
So here I wonder, would it be possible to include more details in the failed batches, so we can easily query it with just 'queryBatches.js'?
Contributor guide
Assessment
This issue has not been assessed yet.