Graylog2 / Graylog2/graylog2-server
Adjust grok() pipeline function when pattern exists but match fails
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
In a pipeline, you may want to do something when grok doesn't match. For example, in one of my pipelines, I'd like to set `gl2_processing_error` when the specified pattern exists but the specified field doesn't match it.
## Expected Behavior
There are two reasonable ways this could work:
1. Add an optional match_required boolean argument to the `grok()` function; when set to true, if the string does not match the pattern, treat as an error much as when the pattern doesn't exist:
```
grok('%{FOO}','bar',false,true) #=> %{FOO} doesn't match 'bar', so an error is raised.
```
2. Add an optional default_result argument; when supplied, if the pattern and string do not match, instead of returning `{}` the function will return the value of the `default_response` argument:
```
grok('%{FOO}','bar',false,{ bing: 'bang' }) #=> %{FOO} doesn't match 'bar', so grok() returns { bing: 'bang' }
```
In this second case, I could pass `{ gl2_processing_error: "Pattern did not match" }` or similar as a default value; if the pattern did not match, then in the subsequent call to `set_fields` the relevant error would be set.
## Current Behavior
If the pattern exists, but the string does not match it, then the `grok()` function returns `{}`. Since the pipeline rule actions have no conditional support, and variables set with let don't persist between stages, the only way to react to a failed match is to store the result of the grok in a field (using `set_field`, not `set_fields`), and then in another stage have a rule like
```
rule "example"
when
has_field('__grok_result') && $message.__grok_result == {}
then
set_field("gl2_processing_error","Didn't match");
remove_field('__grok_result')
end
```
And then you still need another rule to deal get rid of the `__grok_result` field when the match was successful.
## Context
Most of our incoming messages are sent as GELF and never need any grok processing; however, some messages from legacy/third-party apps are send via filebeat and need to be parsed. Filebeat is configured to add a `log_format` field for each file its ingesting, which contains the name of the grok pattern to use to parse it.
If a filebeat configuration specifies a grok pattern that doesn't exist, it results in an error, making it easy to find mistakes by having a dashboard widget showing messages with a gl2_processing_error field; but if the pattern specified by filebeat does exist, but doesn't match the message data, it's much harder for use to detect that we need to update our pattern definition.
## Your Environment
* Graylog Version: 2.5 (planning to move to 3.0, but documented behavior of `grok()` is the same either way)
* Elasticsearch Version: 6.7.1
* MongoDB Version: 2.6.12
* Operating System: CentOS 7
* Browser version: N/A
Contributor guide
Assessment
This issue has not been assessed yet.