Graylog2 / Graylog2/graylog2-server
Support for multifields in ES index/visualizations
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
Allow in visualizations to place field names which are not known to Graylog.
## Why?
There are some restrictions regarding mapping possibilities. In the default mapping for ES, Graylog sets all Fields to "keyword" which in turn leads to long terms in the Lucene index if the information (in this case a long stack trace) is stored in a field other than full_message or message. Max length is hard coded in the Lucene index to 32766 bytes.
A possible solution would be to store almost all fields as "text" if they are from type string as below and store the additional field as keyword.
Mapping in Elasticsearch:
```
[....]
{
"store_generic":{
"mapping":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":1000
}
}
},
"match_mapping_type":"string"
}
}
[....]
```
For the normal query one could use the normal searchbar, with full text
search with terms etc. for all fields (not only message or full message)
In Visualizations one has to use the field appended with an prefix
(.keyword), as it is a subfield in the elasticsearch index, mapped to the
keyword datatype.
For higher performance indexing of the field with a keyword mapping is
limited on fields with fewer than 1000 Characters.
As it stands, here is the following example error output as default.
```
{"type":"illegal_argument_exception","reason":"Document contains at least
one immense term in field=\"response\" (whose UTF8 encoding is longer than
the max length 32766), all of which were skipped. Please correct the
analyzer to not produce such terms. The prefix of the first immense term
is: '[123, 34, 115, 116, 97, 116, 117, 115, 34, 58, 52, 48, 51, 44, 34,
115, 116, 97, 116, 117, 115, 84, 101, 120, 116, 34, 58, 34, 70, 111]...',
original message: bytes can be at most 32766 in length; got
60931","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception:
bytes can be at most 32766 in length; got 60931"}}
```
## Your Environment
See HS-393101859 for additional context.
* Graylog Version:
* Elasticsearch Version:
* MongoDB Version:
* Operating System:
* Browser version:
Contributor guide
Assessment
This issue has not been assessed yet.