elastic / elastic/logstash

From Discuss: No events in DLQ

Open
#7,820 22 comments 0 reactions 0 assignees View on GitHub
discuss DLQ
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

from https://discuss.elastic.co/t/no-events-in-dlq/94309

User is trying to ingest data using the elasticsearch output plugin, and is seeing the following error from Elasticsearch:

```
MapperParsingException: object mapping for [mapping_request.payload] tried to parse field [payload] as object, but found a concrete value
```

but those entries are not being written to the DLQ as expected.

Some messages _have_ been written to the DLQ, but the user is seeing the following errors
```
[2017-07-25T16:41:19,389][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: "/data/logstashData/dead_letter_queue", sincedb_path=>"/data/logstashDLQData/dlq_offset", commit_offsets=>true, codec=>"json_73337b45-c626-4cae-aab9-b2f4d8775a1e", enable_metric=>true, charset=>"UTF-8">, id=>"52ff26acd4e4d46066179db0a3cdf329cbe3c13c-1", enable_metric=>true, pipeline_id=>"main">
```

Contributor guide

Open the contributing guide

Research direction

Start with the linked Discuss report and the dead_letter_queue plugin error, using the Elasticsearch mapping exception as the reproduction symptom. Trace why those failed events are not written to the DLQ and why the plugin restarts; done means the reported failures are handled as expected and the existing DLQ behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.