awslabs / awslabs/aws-lambda-redshift-loader
files not loaded occasionally due "another thread has locked it for loading"
- Dominant language
- JavaScript
- Stars
- 595
- Forks
- 161
- PR merge metrics
- No merged PRs in 30d
Description
A small number of my files (around 10 out of 20k) is not loaded each day, with such error being returned after it hits the timeout:
```
error: The conditional request failed
{
"code": "ConditionalCheckFailedException",
"time": "2021-08-23T21:16:14.506Z",
"requestId": "5UCMPG1CQ6QRQAATPJPH8V536VVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 25.6182288251398
}
```
and logging this:
```
debug: Batch is ready to be processed, but another thread has locked it for loading
```
Despite the debug log above, I wasn't able to find any other batches with these files. My configuration is:
- `batchTimeoutSecs: 240`
- `batchSize: 100`
- `batchSizeBytes: ""`
I wonder if anyone else experienced similar issue and whether there's a simple config tweak that could be applied to mitigate this? Thanks.
Contributor guide
Assessment
This issue has not been assessed yet.