Provide an outgoing buffer mechanism for peer forwarder

Is your feature request related to a problem? Please describe.

The core peer forwarder does not buffer outgoing requests between pipeline batches. It only buffers within a pipeline batch.

An example: Say the incoming buffer's batch_size is set to 50 and there are 5 Data Prepper nodes. Presumably, each batch has 40 requests to send to the 4 peers. Data Prepper will send 4 requests each of size 10.

Describe the solution you'd like

Provide a configuration within Data Prepper to allow peer-forwarding to batch requests before sending them. There would be two configurations:

1. A target batch size. Data Prepper will send this number of messages once enough have accumulated.
2. A timeout for sending. If this time is reached before accumulating enough messages, send the batch data out. This can help with window durations.

Configuration example:

peer_forwarder:
  batch:
    batch_size: 400
    timeout: 2s

When the peer forwarder has received enough messages for the batch_size, it will send exactly the batch_size number of requests. Any other will be batched for the next request. If there are enough left over for another batch_size, these also should be sent immediately.

However, when the timeout is reached for a given batch, that batch will be sent regardless of size. This timeout should start as soon as the batch is started (with the first event added to it).

Additional context

The issue #2147 is possibly caused by too small of batch sizes in HTTP requests.

The requests #2118 could be worked with this task. That is requesting parallel sends rather than sequential. The work probably overlaps.

Start by locating the core peer forwarder and its peer_forwarder configuration, then review the batching requirements and the related issues #2147 and #2118. Done means outgoing requests accumulate to the configured batch size, flush on the configured timeout, and handle leftover messages as described without losing the proposed parallel-send considerations.