elastic / elastic/logstash

Reported behavior (confusing/weird message) is a bug / needs more meaningful messages.

Open
#7,402 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.

For all general issues, please provide the following details for fast resolution:

- Version: Logstash 5.1
- Operating System: RHEL 6.6
- Config File (if you have sensitive info, please remove it):
- Sample Data:
- Steps to Reproduce: Please look at these steps we’ve taken and the exception it generates.

The following config file was used:

```
input {
stdin {
# codec => json
}
}
filter {
# strategy { symbol: xyz parameter { tag: 9001 value: 20170512-13:30:00 } parameter { tag: 9002 value: 20170512-20:00:00 } parameter { tag: 13032 value: 0 } parameter { tag: 9063 value: 0.0 } parameter { tag: 9062 value: 6717.0 } parameter { tag: 9054 value: Hidden_Dark } parameter { tag: 9064 value: 0.0 } parameter { tag: 7128 value: 1494595800000 } parameter { tag: 7129 value: 1494619200000 } parameter { tag: 28079 value: LNDARK } parameter { tag: 13031 value: 100 } parameter { tag: 7148 value: Hidden_Dark } } }
# ADDED FOR TESTING TO SIMULATE 'event' FIELD
grok {
match => {
"message" =>
"^%{GREEDYDATA:event}$"
}
}
grok {
match => {
"event" =>
"strategy { symbol: \S+ %{DATA:strategy_parameters} } }"
}
}
# kv is very limited, we have to make the parsable string as limited
# as kv itself
mutate {
gsub => [
"strategy_parameters", 'parameter \{ tag: ', ' ',
"strategy_parameters", ' value: ', ':',
"strategy_parameters", ' \} ', ''
]
}
# ready for kv
kv {
source => "strategy_parameters"
target => "strategy_parameters"
prefix => "tag_"
field_split => " "
value_split => ":"
}
mutate {
# rename => {"strategy_parameters" => "[event][strategy_parameters]"}
rename => {"strategy_parameters" => "[new_event][strategy_parameters]"}
}
}
output {
stdout { codec => rubydebug }
}
```

We think the crux of the problem is that the customer is carrying around a field called `event` that he continued to parse/extract from - finally getting to the point where he extracted a field called `strategy_parameters` and then re-writes that field by parsing it further using the key-value (kv) function. From here, he tried to assign that new `strategy_parameters` object back into the event field which still has whatever the original value was. We think this is what lines up with the beginning of the stacktrace:

```
Exception in thread "[main]>worker3" java.lang.ClassCastException: expecting List or Map, found class org.logstash.bivalues.StringBiValue
```

LS doesn't like that this new object is being added to the `event` object because they don't match, etc.

In our example, if we leave it as being reassigned to `event` if fails in 5.1 and 5.4 ... but if we add it to a new field called `new_event` then there is no issue - so the suggestion given to the customer would be to use a new field altogether or probably reset/dump `event` if he no longer needs it before.

We know the problem, we know the workaround so the question is now if the feature was designed with such a constraint in mind, or if the behavior is unintended? Or is this a bug?

The outcome here is:

> As it turned out - this is by design in the new Java based event - we got big speed improvements > as well as the binary serializing for the persistent queue but we lost the flexible type system of Ruby.

The issue and ask here is:

> The reported behavior (confusing/weird message) is a bug. The failure is expected, but the message is not useful to users.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.