Graylog2 / Graylog2/graylog2-server

Incomplete documentation for parse_json missing optional depth parameter

Open
#25,108 0 comments 0 reactions 1 assignee Claimed by @tellistone View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior
[Documentation for the pipeline function `parse_json`](https://go2docs.graylog.org/current/making_sense_of_your_log_data/functions_index.html#parse_js) should reflect and explain the current `parse_json` function signature, including the optional `depth` parameter:

Function | Category | Description | Syntax
-- | -- | -- | --
parse_json | String | Parses the value string as JSON, returning the resulting JSON tree. Optionally specify the number of levels to parse. If `depth` > 0, nodes below that level are deleted from the parsed tree. If omitted or set to 0, depth is unlimited.
See also: `to_map` | `parse_json(value: string, [depth: integer])`

Suggestion is based on the current contents of [graylog2-server/src/main/java/org/graylog/plugins/pipelineprocessor/functions/json/JsonParse.java](https://github.com/Graylog2/graylog2-server/blob/dc4f4b3f16715c5b266ab6ec5a99ca93f4500efd/graylog2-server/src/main/java/org/graylog/plugins/pipelineprocessor/functions/json/JsonParse.java)

## Current Behavior
Function | Category | Description | Syntax
-- | -- | -- | --
parse_json | String | Parses the value string as JSON, returning the resulting JSON tree.
See also: `to_map` | `parse_json(value: string)`

## Context
Let's say we're using `parse_json` to handle a variety of incoming logs that contain deeply nested JSON, where we typically only care about extracting data from the top levels to fields. The keys of interest are numerous enough that updating a manually curated list of paths for `select_jsonpath` would be extremely tedious, but parsing and flattening without a depth limit quickly leads to Elastic/OpenSearch exceptions about exceeding total field count limits.

Since https://github.com/Graylog2/graylog2-server/pull/11640 was merged in 2021, `parse_json` has supported optionally limiting `depth`, where conveniently all nodes past that depth are deleted from the parsed tree.

For example, if we want to parse a message containing something like
`{"a": {"b": "c", "d": {"e": {"f": {"g": "h", "i": {"j": "k", "l": "m"}}}}}, "n": "o"}`
and flatten out keys to fields up to a certain depth, we can currently accomplish this with the following pipeline rule where we significantly reduce mapping explosion:
```
let json = parse_json(to_string($message.message), 2); // depth set to 2
let flat_json = flatten_json(to_string(json), "flatten");
let map = to_map(flat_json);
set_fields(map);
```

With no `depth` parameter set, the rule results in

- **a_b**
c
- **a_d_e_f_g**
h
- **a_d_e_f_i_j**
k
- **a_d_e_f_i_l**
m
- **message**
{"a": {"b": "c", "d": {"e": {"f": {"g": "h", "i": {"j": "k", "l": "m"}}}}}, "n": "o"}
- **n**
o

If we set `depth` to 2:

- **a_b**
c
- **message**
{"a": {"b": "c", "d": {"e": {"f": {"g": "h", "i": {"j": "k", "l": "m"}}}}}, "n": "o"}
- **n**
o

There do not seem to be any mentions of this implemented functionality in the docs nor the community forum. Others might also find it very useful if they're dealing with similar usecases where dynamic field promotion is required, and don't want to unnecessarily raise `index.mapping.total_fields.limit`.

## Your Environment

* Graylog Version: 7.0.3
* Graylog Docs Version: 7.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.