Graylog2 / Graylog2/graylog2-server

Allow graylog to store empty strings to elasticsearch

Open
#6,158 2 comments 0 reactions 0 assignees View on GitHub
elasticsearch improvement needs-discussion triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

We are transmitting json formatted log messages to graylog using nxog. On some point we have noted, that graylog stores different number of fields to the elasticsearch after they are parsed by corresponding pipeline. After short investigation we have found that missed fields are empty strings (like in_txt: "" ).

## Expected Behavior
Graylog sends empty strings to elasticsearch. By default custom log fields are not indexed, so it should be Ok from elasticsearch side. String (text) field type does not have null_value option, so its a question if empty string is counted as null value...

## Current Behavior
Graylog does not try to create field with empty string in elasticsearch.

## Possible Solution
Provide possibility to allow empty values. For example as checkbox in index configuration which allows to enable this feature to the custom indexes only in case dynamic templates does not allow creating empty strings. In this case customer will have to create custom index with related mapping to accept empty strings.

## Steps to Reproduce (for bugs)
1. create pipeline with rule to add field using function set_field("empty_string", "");
2. apply pipeline to the steam
3. send message to stream
4. check created fields

## Context
As part of our logging environment we are using [tlog](http://scribery.github.io/tlog/ ) to record user sessions to elasticsearch. There is a tlog-play player tool which allows to play recorded sessions directly from the elasticsearch, but it requires all required fields to be stored in elasticsearch index.

## Your Environment
* Graylog Version: 3.0.2-1
* Elasticsearch Version: 6.8.1
* MongoDB Version: 4.0.10
* Operating System: Debian 9.9

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.