Graylog2 / Graylog2/graylog2-server

processing endless loop in `java.util.regex.Pattern`

Open
#7,289 1 comment 1 reaction 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

This issue is coming from the community ( https://community.graylog.org/t/processing-stuck-with-all-processbufferprocessor-threads-used-up/13558/7 ) and

## Current Behavior
It can happen that GROK/REGEX can send the processing into an endless loop and make Graylog not processing anything. It looks like all processing is stale.

## Steps to Reproduce (for bugs)

Problem can be reproduced:

Use this patterns:

```
"name":"PROFTPDXFER_TIMESTAMP"
"pattern":"%{DAY} %{MONTH} %{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{YEAR}"
"name":"PROFTPD_XFERLOG",
"pattern":"%{PROFTPDXFER_TIMESTAMP:proftpd_xfer_timestamp} %{DATA:proftpd_xfer_transfer-time:int} %{HOSTNAME:proftpd_xfer_remote-host} %{INT:proftpd_xfer_file-size} %{UNIXPATH:proftpd_xfer_filename} %{WORD:proftpd_xfer_transfer-type} %{WORD:proftpd_xfer_action-flag} %{WORD:proftpd_xfer_direction} %{WORD:proftpd_xfer_access-mode} %{USERNAME:proftpd_xfer_username} %{WORD:proftpd_xfer_service-name} %{INT:proftpd_xfer_auth-method} %{DATA:proftpd_xfer_auth-userid} %{WORD:proftpd_xfer_completion-status}"
```

Test the pattern `PROFTPDXFER_TIMESTAMP` with this value:

```
Thu Jan 23 07:51:21 2020 0 1.2.3.4 1234 /ftproot/stuff/Tmp/filexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx{xxxx}xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xml b _ i r abcdef sftp 0 * c
```

This causes the io.krakens.grok.api.Grok.match() to never return. The pattern matching goes into a loop and uses up a cpu core. The actual loop occurs inside java.util.regex.Pattern, so it’s not really a Graylog specifiy problem. I think it’s some kind of bug in the JRE classes. A pattern should either match or not, and not cause the parser to loop forever.

If you have multiple messages in a pipeline that trigger this problem, it will completely halt the pipeline processing in Graylog.

## Context
From time to time we see that user report regular that their Graylog is not processing anymore and a restart solves the problem. Educated guessing will help you to find the problem, but that might not be possible in a multi user/multi admin/multi tier environment.

We need to find a possible way to warn/alert about such or even better ways we are not able to think of currently.

## Your Environment

* Graylog Version: 3.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.