Graylog2 / Graylog2/graylog2-server

Support for multifields in ES index/visualizations

Open
#10,633 0 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?

Allow in visualizations to place field names which are not known to Graylog.

## Why?

There are some restrictions regarding mapping possibilities. In the default mapping for ES, Graylog sets all Fields to "keyword" which in turn leads to long terms in the Lucene index if the information (in this case a long stack trace) is stored in a field other than full_message or message. Max length is hard coded in the Lucene index to 32766 bytes.

A possible solution would be to store almost all fields as "text" if they are from type string as below and store the additional field as keyword.

Mapping in Elasticsearch:
```
[....]
{
"store_generic":{
"mapping":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":1000
}
}
},
"match_mapping_type":"string"
}
}
[....]
```

For the normal query one could use the normal searchbar, with full text
search with terms etc. for all fields (not only message or full message)
In Visualizations one has to use the field appended with an prefix
(.keyword), as it is a subfield in the elasticsearch index, mapped to the
keyword datatype.

For higher performance indexing of the field with a keyword mapping is
limited on fields with fewer than 1000 Characters.

As it stands, here is the following example error output as default.

```
{"type":"illegal_argument_exception","reason":"Document contains at least
one immense term in field=\"response\" (whose UTF8 encoding is longer than
the max length 32766), all of which were skipped. Please correct the
analyzer to not produce such terms. The prefix of the first immense term
is: '[123, 34, 115, 116, 97, 116, 117, 115, 34, 58, 52, 48, 51, 44, 34,
115, 116, 97, 116, 117, 115, 84, 101, 120, 116, 34, 58, 34, 70, 111]...',
original message: bytes can be at most 32766 in length; got
60931","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception:
bytes can be at most 32766 in length; got 60931"}}
```

## Your Environment

See HS-393101859 for additional context.
* Graylog Version:
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.