elastic / elastic/logstash

Missing patterns file causes 100%cpu

Open
#8,464 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Observed that when running logstash on ubuntu, having missing patterns causes logstash to consume 100% cpu indefinitely when restarted. This causes a major problem if the pattern file is not put in place before the config that uses the patterns in it.

Encountered when installing fail2ban configuration.
Example expected patterns file:
```
F2B_DATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[ ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})
F2B_ACTION (\w+)\.(?:\w+)(\s+)?\:
F2B_JAIL \[(?\w+\-?\w+?)\]
F2B_LEVEL (?\w+)\s+
```

A corresponding match action in a config file such as below that uses these:
```
input {
file {
type => "fail2ban"
start_position => "beginning"
path => [ "/var/log/fail2ban.log" ]
}
}

filter {
if [type] == "fail2ban" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => [
"message", "%{F2B_DATE:date} %{F2B_ACTION} %{WORD:level} %{F2B_JAIL} %{WORD:action} %{IP:ip}",
"message", "%{F2B_DATE:date} %{F2B_ACTION} %{F2B_LEVEL} %{GREEDYDATA:msg}?"
]
}

geoip {
source => "ip"
}
}
}
```

Resolution: Change logstash not to consume 100% cpu when encountering tokens like F2B_DATE when that pattern has not been defined; an error should be thrown that the input {} configuration is being ignored because it used an unknown pattern.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.