Graylog2 / Graylog2/graylog2-server
Processing Pipeline `parse_cef` / CEF Input - parse broken message
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## Expected Behavior
When using the `parse_cef` function or the CEF Input it should work for all ingested messages and extract the key-value out of the message.
## Current Behavior
Using `parse_cef` (and ingest CEF messages) takes every `=` and split that into key-value even if the `=` is part of the value of one key:

## Possible Solution
If you use the `key_value` function it works flawless:
key_value(value: to_string(message), trim_value_chars: "\"", trim_key_chars:"\"", delimiters:" ", kv_delimiters:"=");
Maybe the above can be used for the key-value extraction of CEF messages.
```
rule "Isolate CEF"
when
has_field("message")
then
let result = regex(("(CEF:.*)$"),to_string($message.message));
set_field("pure_cef", result["0"]);
end
```
```
rule "Parse CEF"
when
has_field("pure_cef")
then
set_fields(parse_cef(to_string($message.pure_cef), false));
end
```
```
rule "Cleanup CEF"
when
has_field("pure_cef")
then
set_field("message", $message.pure_cef);
remove_field("pure_cef");
end
```
The following message parses cleanly by the default CEF Parserver - no addition `=` is part of the `msg` key
```
CEF:0|Trend Micro Inc.|OSSEC HIDS|v2.9.0|5715|SSHD authentication success.|3|dvc=graylog01 cs1=(srvapp03) 10.130.100.83->/var/log/secure cs1Label=Location classification= syslog,sshd,authentication_success, src=10.130.102.29 shost=10.130.102.29 suser=bogner suser=bogner msg=Mar 15 08:42:07 srvApp03.graylog.lan sshd[40309]: Accepted password for bogner from 10.130.102.29 port 50348 ssh2
```
The following message parses not by the default CEF Parser - because the `msg` field contains `=` in it. The above `key_value` configuration does parese it cleanly.
```
CEF:0|Trend Micro Inc.|OSSEC HIDS|v2.9.0|31104|Common web attack.|6|dvc=graylog01 cs1=(srvws01) 10.130.102.28->/var/log/httpd/access_log cs1Label=Location classification= web,accesslog,attack, src=10.130.104.25 shost=10.130.104.25 msg=10.130.104.25 - - [15/Mar/2019:05:17:21 -0500] "GET /directorypro.cgi?want=showcat&show=../../../../../etc/passwd%00 HTTP/1.1" 302 102 "-" "-"
```
## Context
Graylog does the parsing of CEF Messages right, but the Vendors does not. So we could try to implement the above mentioned fix to be able to parse more messages.
## Your Environment
* Graylog Version: 3.0
* Elasticsearch Version: 6.6
Contributor guide
Assessment
This issue has not been assessed yet.