aws-samples / aws-samples/amazon-elasticsearch-lambda-samples
kinesis -> lambda -> ES only posting some records
- Dominant language
- JavaScript
- Stars
- 386
- Forks
- 170
- PR merge metrics
- No merged PRs in 30d
Description
I setup a simple config to test this out:
logstash-forwarder -> logstash -> kinesis -> lambda -> ES
For batches of size 1, this works fine, log lines are pushed to ES in a few seconds.
For batch size > 1, I am seeing inconsistent behavior, with one of some of my records being pushed. It's often about 5 records that make it through but it's not consistent.
In my lambda logs I sometimes see `Error: Error: socket hang up` a number of times, corresponding to the number of missing records, but this too is inconsistent.
I've confirmed that the expected # of the records appear in the kinesis stream and are transmitted to the lambda job.
What am I doing wrong? Is my kinesis stream too small? My lambda timeout too short?
Contributor guide
Assessment
This issue has not been assessed yet.