awslabs / awslabs/aws-lambda-redshift-loader

files not loaded occasionally due "another thread has locked it for loading"

Open
#233 22 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
595
Forks
161
PR merge metrics
No merged PRs in 30d

Description

A small number of my files (around 10 out of 20k) is not loaded each day, with such error being returned after it hits the timeout:

```
error: The conditional request failed
{
"code": "ConditionalCheckFailedException",
"time": "2021-08-23T21:16:14.506Z",
"requestId": "5UCMPG1CQ6QRQAATPJPH8V536VVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 25.6182288251398
}
```

and logging this:

```
debug: Batch is ready to be processed, but another thread has locked it for loading
```

Despite the debug log above, I wasn't able to find any other batches with these files. My configuration is:
- `batchTimeoutSecs: 240`
- `batchSize: 100`
- `batchSizeBytes: ""`

I wonder if anyone else experienced similar issue and whether there's a simple config tweak that could be applied to mitigate this? Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.