Graylog2 / Graylog2/graylog2-server
Missing check for Grok pattern existence
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Long Grok Patterns can kill processing if there is a wrong field later in the pattern
## Expected Behavior
If any portion of the pattern does not exists then using the Try feature should not return the fields.
## Current Behavior
Currently, in a long Grok Pattern, I have been able to hand the process buffer if there is a pattern missing later in the full pattern
## Possible Solution
Check the Grok parsing engine to correct matching fields or at least fix the try button so that any non-existent pattern causes the full Grok pattern to fail
## Steps to Reproduce (for bugs)
Using the following Pattern on a Firepower Syslog UDP input:
%{DATA:source} %{DATA:dvc} Protocol: %{WORD:proto}, SrcIP: %{IP:src_addr}(, OriginalClientIP: %{DATA:client_ip})?, DstIP: %{IP:dst_addr}, SrcPort: %{INT:src_port}, DstPort: %{INT:dst_port}, TCPFlags: %{BASE16NUM:tcp_flags}, IngressZone: %{DATA:in_zone}, EgressZone: %{DATA:out_zone}, DE: %{DATA:engine}, Policy: %{DATA:policy}, ConnectType: %{WORD:conn_type}, AccessControlRuleName: %{DATA:ac_rule_name}, AccessControlRuleAction: %{WORD:ac_action}, (Prefilter Policy: %{DATA:prefilter_policy},)?( Client: %{WORD:client},)? ApplicationProtocol: %{DATA:app_protocol}, InitiatorPackets: %{INT:init_packets}, ResponderPackets: %{INT:resp_packets}, InitiatorBytes: %{INT:init_bytes}, ResponderBytes: %{INT:resp_bytes},( NAPPolicy: %{DATA:nap_policy},)?( DNSResponseType: %{DATA:dns_resp_type}, Sinkhole: %{DATA:sinkhole}, URLCategory: %{DATA:url_cat}, URLReputation: %{GREEDYDATA:url_rep})
There is not Client: field, yet the try button still allowed for this pattern to complete, but when put in production with the missing Client area the output would slow to <20 messages per second causing the Processing Buffer and the Journal to fill up to 100% each.
## Context
This affected both an older 2.1 version of Graylog and the new 3.0 version I am currently building.
## Your Environment
Logs being forwarded in from a Firepower Management Center to a Graylog 3 server with 8 cores and 64 GB of RAM, 24GB allocated for Elasticsearch and 16 GB allocated for JVM.
* Graylog Version: 3.0
* Elasticsearch Version: 6.6
* MongoDB Version: Current
* Operating System: Debian 9
* Browser version: Chrome (current)
Contributor guide
Assessment
This issue has not been assessed yet.