Graylog2 / Graylog2/graylog2-server

Limit JSON extractor depth

Open
#6,471 0 comments 5 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

The JSON extractor could be used on deeply nested JSON messages to be parsed only up to a certain depth. Say, for a message like:

```
{"a": {"b": {"c": {"d": {"e": "f"}}}}, "g": "h"}
```

It could extract the following keys with a depth of 2:

```
a_b = {"c":{"d":{"e":"f"}}}
g = h
```

## Current Behavior

Currently the JSON extractor extracts all objects to the full depth. For the example above, we'd get

```
a_b_c_d_e = f
g = h
```

## Context

https://community.graylog.org/t/removing-key-from-pipeline-map/12037

I want to parse complex JSON messages that might be deeply nested without creating loads of keys from it. It's specially troublesome for messages that might use id-like keys which will cause the ES index to contain a large number of fields

## Your Environment

* Graylog Version: 3.1.0
* Elasticsearch Version: AWS 6.8
* MongoDB Version: 4.2.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.