Graylog2 / Graylog2/graylog2-server

Add 'match none' match type to pipleine stages

Open
#5,897 0 comments 1 reaction 0 assignees View on GitHub
feature processing triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

I'd like the ability to have a pipeline that proceeds to the next stage only if none of the rules in the rule list matched the message. Effectively, this would turn the following stage into a sort of 'else' clause.

## Expected Behavior
Given the following code:

```
rule "foo is coming"
when
has_field("foo") && $message.foo == "coming"
then
set_field("bar","Hello, World");
end

rule "foo is going"
when
has_field("foo") && $message.foo == "going"
then
set_field("bar","Goodbye, World");
end

rule "default bar"
when
true
then
set_field("bar","Aloha, World");
end

pipeline "example"
stage 0 match none
rule "foo is coming";
rule "foo is going";
stage 1 match any
rule "default foo";
end
```

A message whose `foo` field is neither "coming" nor "going" should have its `bar` field set to "Aloha, World".

## Current Behavior
The above pipeline is a syntax error presently. In order to achieve the desired behavior, the `default bar` rule and the the pipeline must be modified like so:

```
rule "default bar"
when
!($message.foo == "coming" || $message.foo == "going")
then
set_field("bar","Aloha, World");
end

pipeline "example"
stage 0 match any
rule "foo is coming";
rule "foo is going";
rule "default foo";
end
```

This is simple enough in this limited example case, but every time you add a new rule with a different possible value for `foo`, you also have to edit the `default foo` rule to exclude that value; and if the condition is more complex than simply checking the equality of one field to a string literal, you have to duplicate that complexity, and waste cycles evaluating it twice for each message.

An alternative would be to have a stage -1, which unconditionally adds a `_example_pipeline_dummy_field` to every message; have `foo is coming` and `foo is going` remove that dummy field after performing their other processing; add a plain `when true then end` rule to stage 0 so that stage 1 would be reached even though none of the positive conditions matched; and then have the `default foo` rule in stage 1 check for the presence of the `_example_pipeline_dummy_field` field. This _works_, but it adds unnecessary complexity to setting up a pipeline and makes them harder to reason about.

## Context
We have a few pipelines where we use a number of rules in a stage, each of which checks a specific fields against various possibilities and behaves accordingly. The conditions are defined in such a way that no two rules will ever match, but it _is_ possible for _none_ of the rules to match. Defining how to behave in that case is unnecessarily cumbersome.

## Your Environment
* Graylog Version: 2.5.1
* Elasticsearch Version: 6.7.1
* MongoDB Version: 2.6.12
* Operating System: CentOS 7
* Browser version: N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.