Graylog2 / Graylog2/graylog2-server
SYSLOGBASE Grok pattern creates invalid "timestamp" value
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
Using the `SYSLOGBASE` Grok pattern should create a valid `timestamp` field value in the message object.
## Current Behavior
The `SYSLOGBASE` Grok pattern is creating the `timestamp` value as `String` type instead of a date type.
This is the current definition of the `SYSLOGBASE` Grok pattern:
```
%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
```
The `SYSLOGTIMESTAMP` value will be written as `timestamp` in the message. Since there is no `date` converter on that pattern, the value type will be `String`.
In `Message#toElasticsearchObject()` we handle invalid `timestamp` fields by trying to parse the strings. If that fails, we use the current timestamp as `timestamp` value as a fallback.
## Possible Solution
One possible solution could be to modify the `SYSLOGBASE` pattern to include a `date` converter for the `SYSLOGTIMESTAMP` pattern. We do the same for parsing Apache log timestamps:
```
%{HTTPDATE:timestamp;date;dd/MMM/yyyy:HH:mm:ss Z}
```
The problem is, that we might need more than one date format to parse syslog dates. For single digit days we need `MMM d HH:mm:ss yyyy` and for multi digit days `MMM dd HH:mm:ss yyyy`.
Not sure if we can construct a single date format that can handle both.
Another way to fix this would be to adjust the Grok library we are using (we maintain a fork) and extend it to handle multiple date formats. Alternatively we can implement a different date converter like `syslog_date` to handle this in Java code.
If we fix this, we need to update `V20191121145100_FixDefaultGrokPatterns` to make sure the fixed grok pattern is updated in existing systems.
## Steps to Reproduce (for bugs)
1. Build an extractor or pipeline rule to parse syslog messages with a Grok pattern
2. Inspect the `timestamp` field
3. Check that indexed timestamp of the message isn't the same as the syslog timestamp (it will be the current time)
## Context
Existing issues running into a similar problem:
- https://github.com/Graylog2/graylog2-server/issues/1577
- https://github.com/Graylog2/graylog2-server/issues/1656
- https://github.com/Graylog2/graylog2-server/issues/7363
## Your Environment
* Graylog Version: 3.2.0
Contributor guide
Assessment
This issue has not been assessed yet.