Graylog2 / Graylog2/graylog2-server

Missing check for Grok pattern existence

Open
#5,731 0 comments 0 reactions 0 assignees View on GitHub
bug processing triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Long Grok Patterns can kill processing if there is a wrong field later in the pattern

## Expected Behavior
If any portion of the pattern does not exists then using the Try feature should not return the fields.

## Current Behavior
Currently, in a long Grok Pattern, I have been able to hand the process buffer if there is a pattern missing later in the full pattern

## Possible Solution
Check the Grok parsing engine to correct matching fields or at least fix the try button so that any non-existent pattern causes the full Grok pattern to fail

## Steps to Reproduce (for bugs)
Using the following Pattern on a Firepower Syslog UDP input:
%{DATA:source} %{DATA:dvc} Protocol: %{WORD:proto}, SrcIP: %{IP:src_addr}(, OriginalClientIP: %{DATA:client_ip})?, DstIP: %{IP:dst_addr}, SrcPort: %{INT:src_port}, DstPort: %{INT:dst_port}, TCPFlags: %{BASE16NUM:tcp_flags}, IngressZone: %{DATA:in_zone}, EgressZone: %{DATA:out_zone}, DE: %{DATA:engine}, Policy: %{DATA:policy}, ConnectType: %{WORD:conn_type}, AccessControlRuleName: %{DATA:ac_rule_name}, AccessControlRuleAction: %{WORD:ac_action}, (Prefilter Policy: %{DATA:prefilter_policy},)?( Client: %{WORD:client},)? ApplicationProtocol: %{DATA:app_protocol}, InitiatorPackets: %{INT:init_packets}, ResponderPackets: %{INT:resp_packets}, InitiatorBytes: %{INT:init_bytes}, ResponderBytes: %{INT:resp_bytes},( NAPPolicy: %{DATA:nap_policy},)?( DNSResponseType: %{DATA:dns_resp_type}, Sinkhole: %{DATA:sinkhole}, URLCategory: %{DATA:url_cat}, URLReputation: %{GREEDYDATA:url_rep})

There is not Client: field, yet the try button still allowed for this pattern to complete, but when put in production with the missing Client area the output would slow to <20 messages per second causing the Processing Buffer and the Journal to fill up to 100% each.

## Context
This affected both an older 2.1 version of Graylog and the new 3.0 version I am currently building.

## Your Environment
Logs being forwarded in from a Firepower Management Center to a Graylog 3 server with 8 cores and 64 GB of RAM, 24GB allocated for Elasticsearch and 16 GB allocated for JVM.

* Graylog Version: 3.0
* Elasticsearch Version: 6.6
* MongoDB Version: Current
* Operating System: Debian 9
* Browser version: Chrome (current)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.