Graylog2 / Graylog2/graylog2-server

Make graylog install custom index mappings into ES for standard field types

Open
#4,801 2 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

https://github.com/Graylog2/graylog2-server/issues/3603 looks like a similar problem that probably also could be avoided by (pre)setting the index format

## Expected Behavior
Don't allow Elasticsearch indices index format to be guessed into something that breaks future log processing

## Current Behavior
Then the first message is written to a new indices, Elasticsearch guesses the index format based on what comes in. In most cases it works well, but sometimes tricky logs can make elasticsearch guess the format wrong. Then you typically get messages like in server.log:
https://community.graylog.org/t/error-messages-failed-to-index-499-messages-please-check-the-index-error-log-in-your-web-interface-for-the-reason-error-one-or-more-of-the-items-in-the-bulk-request-failed-check-bulkresult-getitems-for-more-information/5304

## Possible Solution
For the syslog supplied fields [application_name, facility, level, process_id] it would be good if the field type could be enforced by adding a custom index mapping at (graylog) installation/initiation time.

## Steps to Reproduce (for bugs)

## Context
It would make the default graylog installation less error prune to syslog messages that are formatted in a sloppy way.

## Your Environment

* Graylog Version: 2.4.4
* Elasticsearch Version: 5.6.9
* MongoDB Version: 2.6.10
* Operating System: Ubuntu 16.04

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.