Introduce the concept of topics to the DLQ
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
During the SupportSummit developer take-over, I handled two tickets where users wanted to get DLQ entries off disk and into ES ASAP - both users fell short with simple dlq input -> es output configs because the entries had different shapes for one mapping.
**I am proposing that we generate a "topic" that categorises the entry so that dlq ingest pipelines can be built around specific topics to transform them and send onward if the original fault can't be fixed. These categories should be rich in content, e.g. "output/elasticsearch/mapper_parsing_exception/failed to parse/\/number_format_exception". Design wise, a tagging mechanism [@metadata][dead_letter_queue][dlq_tags] is a probably more flexible. With a topic generated by `event.get("[@metadata][dead_letter_queue][dlq_tags]").join('/')` or similar.**
General pitfalls:
- The entires cannot be indexed directly unless they all have the same shape.
- The differently shaped entries can't be transformed into a conformal shape for insertion into the original index because information is missing.
- As more plugins get given the ability to write events to the DLQ, the shape of the event will be more different still - as will the variations of the `reason` value.
General problems to solve (see sample entry below):
- While we do have the plugin_type and id, the original target destination for this LS event is lost. The destination should include address/port, index/topic. Some of this may be alleviated by a reference to a centralised config record.
- The actual problem that this event encountered, `mapper_parsing_exception` + `failed to parse []` + `number_format_exception`, is buried deep in the response from ES.
Sample DLQ metadata entry:
```
"@metadata" => {
"dlq_message" => "true",
"dead_letter_queue" => {
"plugin_type" => "elasticsearch",
"plugin_id" => "08645c9aa7d0c1e40a36b2c83adc6bbd40b1c253-11",
"reason" => "Could not index event to Elasticsearch. status: 400, action: [\"index\",
{:_id=>nil, :_index=>\"logstash-logs-2017.09.28\", :_type=>\"logs\", :_routing=>nil},
2017-09-28T23:54:09.258Z %{host} %{message}], response: {\"index\"=>
{\"_index\"=>\"logstash-logs-2017.09.28\", \"_type\"=>\"logs\", \"_id\"=>
\"AV7K6W7-8Vei4cyktbkL\", \"status\"=>400, \"error\"=>
{\"type\"=>\"mapper_parsing_exception\",
\"reason\"=>\"failed to parse [mysecondfield]\",
\"caused_by\"=>
{\"type\"=>\"number_format_exception\",
\"reason\"=>\"For input string: \\\"abc123\\\"\"}}}}",
"entry_time" => #
}
}
```
Other Issues:
https://github.com/elastic/logstash/issues/8242
Contributor guide
Assessment
This issue has not been assessed yet.