Graylog2 / Graylog2/graylog2-server

Indexer failures should produce more information for root cause analysis

Open
#19,615 3 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?

Indexer failure messages in the UI look something like this:

2 hours ago | techlog_52 | c5f5e982-287f-11ef-954a-00505687ab33 | OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [level] of type [long] in document with id 'c5f5e982-287f-11ef-954a-00505687ab33'. Preview of field's value: 'Information']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=For input string: "Information"]];
-- | -- | -- | --

This is not really helpful for resolving the issue. If you have large amount of servers, systems, and components, the issue could be in numerous components generating logs, different responsible teams and so on. It is impossible to start diagnostics when you don't even know whom to start it with.

It seems OpenSearch doesn't log the issue from the example message I provided at all. It would apparently require debug logging level to appear, and that is simply not doable when you receive huge volume of logs. Graylog should be the component that produces extra information.

Alternatives:
- Add sender's IP to the indexer failure messages and UI (probably enough, somewhat easy to implement)
- Add logging of the message to the server logs (probably easy, and also enough for system admins)
- Revive the dead letter implementation (complex, most convenient for system admins)

See MessagesAdapterOS2 for clues. Offending message at least should be available in most cases.

## Why?

The current indexer failures view doesn't provide basic required information for resolving the issues. It is not possible to resolve indexer failures in more complex environments.

## Your Environment

n/a

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.