aws-samples / aws-samples/amazon-elasticsearch-lambda-samples

kinesis -> lambda -> ES only posting some records

Open
#1 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
386
Forks
170
PR merge metrics
No merged PRs in 30d

Description

I setup a simple config to test this out:

logstash-forwarder -> logstash -> kinesis -> lambda -> ES

For batches of size 1, this works fine, log lines are pushed to ES in a few seconds.

For batch size > 1, I am seeing inconsistent behavior, with one of some of my records being pushed. It's often about 5 records that make it through but it's not consistent.

In my lambda logs I sometimes see `Error: Error: socket hang up` a number of times, corresponding to the number of missing records, but this too is inconsistent.

I've confirmed that the expected # of the records appear in the kinesis stream and are transmitted to the lambda job.

What am I doing wrong? Is my kinesis stream too small? My lambda timeout too short?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.