awslabs / awslabs/lambda-streams-to-firehose
Exceeding 4MB limit
- Dominant language
- JavaScript
- Stars
- 278
- Forks
- 90
- PR merge metrics
- No merged PRs in 30d
Description
We're running 1.4.5.
We're commonly getting errors for exceeding the 4MB write limit to Firehose. This causes the lambda to retry indefinitely on the same data, causing processing on the Kinesis shard to backup until the data ages out.
We solved this by editing constants.js:
```
// firehose max PutRecordBatch size 4MB
FIREHOSE_MAX_BATCH_BYTES = 4 * 1024 * 1024;
```
Is now:
```
// firehose max PutRecordBatch size 2MB
FIREHOSE_MAX_BATCH_BYTES = 2 * 1024 * 1024;
```
And for good measure we also did the following:
```
FIREHOSE_MAX_BATCH_COUNT = 500;
```
Is now:
```
FIREHOSE_MAX_BATCH_COUNT = 250;
```
The above changes immediately solved our issue. Looking through the code, I'm not sure how a batch of > 4MB is able to get through, but it appears that was the case for us.
Contributor guide
Assessment
This issue has not been assessed yet.