S3 Input Filter Using S3 Event Notifications
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
## The Problem - Slow S3 input
The S3 input filter is pretty awesome, but we've found that for buckets with high write throughput, there's a couple of drawbacks:
1. _Clustering isn't an option_ - We like to feed most of our logstash instances from some sort of message queue so we can have redundancy and scale horizontally. Multiple instances of logstash reading from the same S3 bucket will read the same data and clustering isn't that useful
2. _Listing the S3 bucket takes a long time_ - The S3 input filter winds up taking a lot of time listing and re-listing the bucket contents for buckets with a high # of new objects. The input filter can't keep up with the activity in the bucket.
## A Solution - S3 Event Notifications via SQS
Seems like an elegant way to solve these problems would be to use [S3 Event Notifications](http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) and let SQS tell the input filter when there's new objects to read. The input filter parses the notification and grabs the contents of the S3 object.
1. _Clustering is easy using SQS_ - We use a message queue. Each notification is delivered to a single logstash instance. World peace ensues.
2. _No more time spent listing the bucket_ - S3 event notifications tell us where the new data is. No need to do listing.
## What's the Best Implementation?
We haven't contributed to logstash before and would like to get opinions on what the best way to implement this is. A couple of options:
1. _A totally new `s3notify` input filter_ - We've got a basic proof of concept. We could just go make a new filter.
2. _Derive from the existing SQS input filter_ - The [sqs input filter](https://github.com/elastic/logstash/blob/v1.4.2/lib/logstash/inputs/sqs.rb) is nice and does smart stuff, like progressive backoff and a reasonable sleep between API calls. Seems like we should reuse that. We could move [this block of the `run` method](https://github.com/elastic/logstash/blob/v1.4.2/lib/logstash/inputs/sqs.rb#L118-130) into its own method and then override it in a `s3notify` input filter. The benefit here is that we're reusing as much code as possible and fixes to the existing sqs input filter will improve `s3notify`.
3. _Add an `s3notify` config value to the existing sqs input filter_ - If we think that reading S3 event notification from an SQS queue and then retrieving the contents of S3 objects is a primary use case for the SQS input filter, then this could become an optional behavior of the exiting SQS filter. No need to create a new filter, just add this behavior.
thanks for reading this far from @ivanlei & @tomelm
Contributor guide
Assessment
This issue has not been assessed yet.