Big batch size reduces latency of an event appearing in Elastic Search

- Version: logstash 2.3.1
- Operating System: Linux 4.3.0

We are using ELK to collect production logs. We use rsyslogd as a way to get these logs to logstash.
We expect logs to appear in Elastic Search in a few seconds after they are written to syslog.
Our log rate is often steady, but sometimes system can generate A LOT of logs.

When this happens we see that rsyslogd -> logstash queue size increases because logstash is very slow in processing these messages. According to processing rate, logstash will process all queued messages in few hours. It is unacceptable.

We tried increasing --pipeline-batch-size from 5000 to 50000 and queue vanished in a minute.

But leaving --pipeline-batch-size value as 50000 is unacceptable too because of this:

> The maximum time that Logstash waits between receiving an event and processing that event in a filter is the product of the pipeline_batch_delay and pipeline_batch_size settings.

We are getting huge latency from message creation to it appearing in Elastic Search (50000 \* 5 = 250000 milliseconds = 250 seconds).

It is very strange decision. When one hears the names of variables pipeline_batch_delay and pipeline_batch_size it expects logstash to wait for 50000 messages OR 5 ms passing before sending data for processing. But not product of these two values.
1. I propose that batch be sent forward when pipeline_batch_size messages come OR pipeline_batch_delay passes.
2. If pipeline_batch_delay is used internally for something else, I propose some new variable with same meaning.
3. If both 1 and 2 are not acceptable for you, what is your advice for being able to process huge numbers of messages efficiently and keeping latency small?

Thank you!