awslabs / awslabs/lambda-streams-to-firehose

Exceeding 4MB limit

Open
#49 7 comments 0 reactions 1 assignee Claimed by @IanMeyers View on GitHub
Dominant language
JavaScript
Stars
278
Forks
90
PR merge metrics
No merged PRs in 30d

Description

We're running 1.4.5.

We're commonly getting errors for exceeding the 4MB write limit to Firehose. This causes the lambda to retry indefinitely on the same data, causing processing on the Kinesis shard to backup until the data ages out.

We solved this by editing constants.js:

```
// firehose max PutRecordBatch size 4MB
FIREHOSE_MAX_BATCH_BYTES = 4 * 1024 * 1024;
```

Is now:

```
// firehose max PutRecordBatch size 2MB
FIREHOSE_MAX_BATCH_BYTES = 2 * 1024 * 1024;
```

And for good measure we also did the following:

```
FIREHOSE_MAX_BATCH_COUNT = 500;
```

Is now:

```
FIREHOSE_MAX_BATCH_COUNT = 250;
```

The above changes immediately solved our issue. Looking through the code, I'm not sure how a batch of > 4MB is able to get through, but it appears that was the case for us.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.