Graylog2 / Graylog2/graylog2-server

Performance Improvement: Combine Regex in Sigma Rules

Open
#25,990 0 comments 1 reaction 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Sigma rules are often slow to evaluate because they use a lot of "OR" terms. Using the regex-native OR operator leads to large performance improvements.

## What?

Sigma rules lead to queries containing many terms combined by OR. For example:

```
title: Process Command Line Contains Any Letter a-z
logsource:
category: process_creation
product: windows
detection:
selection:
CommandLine|contains:
- 'a'
- 'b'
- 'c'
...
- 'x'
- 'y'
- 'z'
condition: selection
```

Graylog translates this into:

```
process_command_line:(/.*a.*/ OR /.*b.*/ OR /.*c.*/ OR /.*d.*/ OR /.*e.*/ OR /.*f.*/
OR /.*g.*/ OR /.*h.*/ OR /.*i.*/ OR /.*j.*/ OR /.*k.*/
OR /.*l.*/ OR /.*m.*/ OR /.*n.*/ OR /.*o.*/ OR /.*p.*/
OR /.*q.*/ OR /.*r.*/ OR /.*s.*/ OR /.*t.*/ OR /.*u.*/
OR /.*v.*/ OR /.*w.*/ OR /.*x.*/ OR /.*y.*/ OR /.*z.*/)
```

A much more performant translation is this, however:

```
process_command_line:/.*(a|b|c|d|e|f|g|h|i|j|k|l|m|n|o|p|q|r|s|t|u|v|w|x|y|z).*/
```

## Why?

Time measurements:

OR-query: 14 seconds
regex query: 4 seconds

This matters for:

- general resource usage of Graylog
- every time an alert is investigated the analysts must wait for the query to load

## Your Environment

* Graylog Version: 7.1.0+549ae7f
* OpenSearch Version: graylog-datanode 7.1.0+549ae7f
* MongoDB Version: irrelevant
* Operating System: Linux
* Browser version: irrelevant

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.