Graylog2 / Graylog2/graylog2-server

SYSLOGBASE Grok pattern creates invalid "timestamp" value

Open
#7,375 1 comment 0 reactions 0 assignees View on GitHub
bug processing triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## Expected Behavior

Using the `SYSLOGBASE` Grok pattern should create a valid `timestamp` field value in the message object.

## Current Behavior

The `SYSLOGBASE` Grok pattern is creating the `timestamp` value as `String` type instead of a date type.

This is the current definition of the `SYSLOGBASE` Grok pattern:

```
%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
```

The `SYSLOGTIMESTAMP` value will be written as `timestamp` in the message. Since there is no `date` converter on that pattern, the value type will be `String`.

In `Message#toElasticsearchObject()` we handle invalid `timestamp` fields by trying to parse the strings. If that fails, we use the current timestamp as `timestamp` value as a fallback.

## Possible Solution

One possible solution could be to modify the `SYSLOGBASE` pattern to include a `date` converter for the `SYSLOGTIMESTAMP` pattern. We do the same for parsing Apache log timestamps:

```
%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:mm:ss Z}
```

The problem is, that we might need more than one date format to parse syslog dates. For single digit days we need `MMM d HH:mm:ss yyyy` and for multi digit days `MMM dd HH:mm:ss yyyy`.

Not sure if we can construct a single date format that can handle both.

Another way to fix this would be to adjust the Grok library we are using (we maintain a fork) and extend it to handle multiple date formats. Alternatively we can implement a different date converter like `syslog_date` to handle this in Java code.

If we fix this, we need to update `V20191121145100_FixDefaultGrokPatterns` to make sure the fixed grok pattern is updated in existing systems.

## Steps to Reproduce (for bugs)

1. Build an extractor or pipeline rule to parse syslog messages with a Grok pattern
2. Inspect the `timestamp` field
3. Check that indexed timestamp of the message isn't the same as the syslog timestamp (it will be the current time)

## Context

Existing issues running into a similar problem:

- https://github.com/Graylog2/graylog2-server/issues/1577
- https://github.com/Graylog2/graylog2-server/issues/1656
- https://github.com/Graylog2/graylog2-server/issues/7363

## Your Environment

* Graylog Version: 3.2.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.