Graylog2 / Graylog2/graylog2-server
Limit JSON extractor depth
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
The JSON extractor could be used on deeply nested JSON messages to be parsed only up to a certain depth. Say, for a message like:
```
{"a": {"b": {"c": {"d": {"e": "f"}}}}, "g": "h"}
```
It could extract the following keys with a depth of 2:
```
a_b = {"c":{"d":{"e":"f"}}}
g = h
```
## Current Behavior
Currently the JSON extractor extracts all objects to the full depth. For the example above, we'd get
```
a_b_c_d_e = f
g = h
```
## Context
https://community.graylog.org/t/removing-key-from-pipeline-map/12037
I want to parse complex JSON messages that might be deeply nested without creating loads of keys from it. It's specially troublesome for messages that might use id-like keys which will cause the ES index to contain a large number of fields
## Your Environment
* Graylog Version: 3.1.0
* Elasticsearch Version: AWS 6.8
* MongoDB Version: 4.2.0
Contributor guide
Assessment
This issue has not been assessed yet.