Graylog2 / Graylog2/graylog2-server

Improvements for Grok Debugging within Graylog

Open
#14,451 0 comments 2 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?

Grok is a very powerful solution for parsing logs without the barrier of learning regular expressions. There are some examples where this can be challenging to be performed in Graylog.

## An example custom Grok pattern

![image](https://user-images.githubusercontent.com/1891183/212719299-3fbe9e45-2789-412d-8f33-0b072243950a.png)

### Improvement 1: The Grok debugger tool should reflect the syntax used by Graylog

When escaping is required and not required is very unclear in Graylog GROK. Square brackets are the most common cause of grief. Here is a rule using exactly the copy-pasted pattern above, which parsed successfully in the Pattern editor:
![image](https://user-images.githubusercontent.com/1891183/212715964-0c4597f5-0c51-4b77-9800-bcc4f16ff5f6.png)

### Improvement 2: Reflect the grok options used by Graylog
There is no way to indicate: "Named Captures Only" while using the pattern editor, so you often see many subpatterns you may not wish to see and in longer, complicated logs this can get difficult.

### Improvement 3: Searching Filter pattern only shows the top 8 results and they're double-spaced (can't search for 'word' or 'greedy')

### Improvement 4: Not having to arbitrarily name something just to do a debug test to build a rule

Just for comparison's sake - here is the same function in Kibana:
![image](https://user-images.githubusercontent.com/1891183/212717663-20d27dc4-f21d-42b9-8360-83d32974369b.png)

### Improvement 5: Word wrap in Pipeline rules

Without this - any grok pattern longer than 100 characters (most of them) become difficult to edit in the browser. Most of those I've conversed with use a workflow of 'Code Editor : Build & Debug -> Rule editor: Additional Build & Debug' so they can see what they're doing, but obviously there is no way of syntax-checking a pipeline rule in a code editor.

### Improvement 6: Allow multiline sample data

Its a bit misleading that it allows, and does not produce error or warning, multiple lines of log data but only parses the first. This behaviour initially leads the user to believe you're not matching the second line of log data. I have certainly spent time debugging a pattern wondering why I wasn't matching a second line.

# Thanks for your time.

## Why?

Currently, with certain online tools having gone offline recently - it is now a rather tedious process to create Grok patterns.

These are only suggestions - I would just love to see an improvement to using Grok in Graylog as its a very powerful way of extracting data and currently there are some challenges to realizing that power.

## Your Environment

* Graylog Version: Cloud 5.0.0 - but exact same as self-hosted
* Elasticsearch Version: ??
* MongoDB Version:??
* Operating System: ??
* Browser version:

![image](https://user-images.githubusercontent.com/1891183/212720744-21a43429-ec0e-40e2-99aa-d25be89ee94b.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.