Graylog2 / Graylog2/graylog2-server

Support iteration in pipeline rules

Open
#3,605 4 comments 4 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

This is a feature request pertaining to the capabilities of pipeline rules. The suggestion is to add support for looping over Map results in order to cover important use cases for functions returning them like grok() or key_value().

## Current situation

Pipeline rules are limited in that iteration over Map objects (key-vector, value-vector, key-value, you name it) as returned by key_value() or grok() is not supported. This hinders some useful applications of these filters.

Given some key-value pairs processed with key_value(), for example sudo syslog output as below,
```
<85>1 2017-03-13T16:13:27.505273+01:00 rt-x-y sudo - - - root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/ls
```
we want to extract the details about the executed command from this message and index it into elasticsearch:
```
TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/ls
```
If we use
```
rule "syslog:sudo:capture command info"
when
contains(to_string($message.message), "COMMAND=") == true
then
let fields = key_value(to_string($message.message), " ;:");
set_fields(fields);
end
```
we index
```
TTY=pts/0
PWD=/root
USER=root
COMMAND=/bin/ls
```
with uppercase keys, which in most cases should hurt consistency as most people probably prefer lowercase keys. This also makes search queries like `_exists_:command` return no results.

If we just do
```
key_value(lowercase(to_string($message.message)), " ;:")
```
instead, this fixes our key problem but lowercases the value parts as well, which might lead to information loss as command names for example are typically case-sensitive (in this special case).

## Possible solution

Support looping constructs inside pipeline rules, for maps. So we could e.g. have:
```
rule "syslog:sudo:capture command info"
when
contains(to_string($message.message), "COMMAND=") == true
then
let fields = key_value(to_string($message.message), " ;:");
foreach( [key, value] in fields ) {
let key = lowercase(key);
set_field(key, value);
}
end
```
or even prefix the keys:
```
foreach( [key, value] in fields ) {
let key = concat("sudo_", lowercase(key));
set_field(key, value);
}
```

Using a complete embedded language for pipeline rules would probably make things simpler.

## Your Environment

* Graylog Version: 2.2.2-1
* Elasticsearch Version: 2.4.4
* MongoDB Version: 3.4.2
* Operating System: Debian 8.7
* Browser version: Firefox 52.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.