Sending DLQ messages to disk not optimal in all cases

Logstash: 5.6

Currently it seems the only option for the dead letter queue is to write the files to disk. This option works well if you're running instances of Logstash on VMs (with persistent drives) that aren't within groups that are auto-scallable although doesn't tend to work as well in scenarios where someone might be using Logstash in Kubernetes auto-scaled containers without persistent volume claims. You could end up either loosing events from the DLQ but also you can't evenly distribute the task of processing/indexing DLQ events to other logstash instances evenly.

For example, If you end up having a single pipeline that indexes a large volume of events and also causes a large amount of mapping conflicts, you'll be considerably increasing the load on that instance, and you can't share the task of processing its generated DLQ. Considering this, it would be great if an option could be added to publish DLQ messages to a Kafka topic directly. This way, many logstash instances could subscribe to this topic via the Kafka input and then evenly share the task of processing/indexing the messages.

Start by examining the existing disk-backed dead letter queue behavior and how Logstash's Kafka input consumes a topic. Define the Kafka publishing option and confirm that multiple Logstash instances can share processing of the generated DLQ messages without losing them in auto-scaled environments.

Written by the indexing model from the issue text.