elastic / elastic/logstash

split filter is not working as expected - split a json array/array

Open
#3,611 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

I have a JSON-Message with an array in an array. I want to split that into multiple events. This is my configuration:

```
input {
stdin {
codec => "json"
}
}

filter {
split {
field => "bean"
}
if [bean][method] {
split {
field => "bean[method]"
}
}
}
```

first array with 2 items, inner array 2 items:

```
{"foo": "someValue","bean": [
{"name": "bean1Name", "method": [{"name": "bean1meth1","max": 5},{"name": "bean1meth2","max": 3}]},
{"name": "bean2Name", "method": [{"name": "bean2meth1","max": 0},{"name": "bean2meth2","max": 1}]}]
}
```

all fine as I need it:

```
{"foo":"someValue","bean":{"name":"bean1Name","method":{"name":"bean1meth1","max":5}},"@version":"1",...
{"foo":"someValue","bean":{"name":"bean1Name","method":{"name":"bean1meth2","max":3}},"@version":"1",...
{"foo":"someValue","bean":{"name":"bean2Name","method":{"name":"bean2meth1","max":0}},"@version":"1",...
{"foo":"someValue","bean":{"name":"bean2Name","method":{"name":"bean2meth2","max":1}},"@version":"1",...
```

Now the problems, the first array has two items, inner array just one:

```
{"foo": "someValue","bean": [
{"name": "bean1Name", "method": [{"name": "bean1meth1","max": 5}]},
{"name": "bean2Name", "method": [{"name": "bean2meth1","max": 0}]}]
}
```

As you can see, the method is written as a list again. Not what I want:

```
{"foo":"someValue","bean":{"name":"bean1Name","method":[{"name":"bean1meth1","max":5}]},"@version":"1",...
{"foo":"someValue","bean":{"name":"bean2Name","method":[{"name":"bean2meth1","max":0}]},"@version":"1",...
```

next problem, the first array has one item:

```
{"foo": "someValue","bean": [
{"name": "bean1Name", "method": [{"name": "bean1meth1","max": 5}]}]
}
```

That is'n working at all:

```
Only String and Array types are splittable. field:bean[method] is of type = Java::JavaUtil::LinkedHashMap> ... split.rb:46:in `filter'
```

I use Logstash 1.5.2

Contributor guide

Open the contributing guide

Research direction

Start with the nested split filter configuration and the reported split.rb:46 location. Run the three JSON examples through Logstash 1.5.2 and compare how singleton inner arrays are represented with the expected events. Done means nested arrays produce consistent object values instead of singleton lists or a type error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.