fluent / fluent/fluentd

Max nesting level for json parser

Open
#3,311 9 comments 3 reactions 1 assignee Claimed by @ashie View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

**Is your feature request related to a problem? Please describe.**
I want to have option for the json parser plugin to limit nesting level for the parsing.
My developers send huge metadata json, after parsing it "eats" elasticsearch fields.

```
curl --progress-bar "http://127.0.0.1:9200/index-logs1/_field_caps?fields=*" | jq '.fields' | grep -E '^ "' 2>/dev/null | grep metadata | awk -F. '{print NF}' | sort -n | wc -l
733

curl --progress-bar "http://127.0.0.1:9200/index-logs1/_field_caps?fields=*" | jq '.fields' | grep -E '^ "' 2>/dev/null | grep context | awk -F. '{print NF}' | sort -n | wc -l
218
```

But if i could limit nesting level for parsing, it would dramatically decreased fields count:
```
curl --progress-bar "http://127.0.0.1:9200/index-logs1/_field_caps?fields=*" | jq '.fields' | grep -E '^ "' 2>/dev/null | grep metadata | awk -F. 'NF>5 {print NF}' | sort -n | wc -l
101

curl --progress-bar "http://127.0.0.1:9200/index-logs1/_field_caps?fields=*" | jq '.fields' | grep -E '^ "' 2>/dev/null | grep context | awk -F. 'NF>5 {print NF}' | sort -n | wc -l
25
```

**Describe the solution you'd like**
Set parameter to json parser section - `max_nesting(int)`
So the parser would leave unparsed json after the nesting is reacher.

**Describe alternatives you've considered**
-

**Additional context**
As i can see, parameter support with main json ruby libraries:
- [oj](https://github.com/ohler55/oj/blob/develop/pages/Options.md#max_nesting)
- [yajl](https://github.com/brianmario/yajl-ruby/blob/0277f32acc2f7a0443a36eb6e47c58259d396e8c/benchmark/parse_json_and_yaml.rb#L34)
- [json](https://ruby-doc.org/stdlib-2.5.3/libdoc/json/rdoc/JSON.html#parse-method)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.