Graylog2 / Graylog2/graylog2-server
Incorrectly parsed CEF formatted messages containing mixed case field names
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
A CEF formatted message containing field name such as _cs1Label_ is not parsed correctly.
In case the field name is renamed to lowercase, e.g. _cs1label_, everything works as expected.
I have to deal with a commercial firewall which produces CEF messages containing key-value pairs such as _cs1Label=192.168.1.1_.
In the Elasticsearch index mapping, I get the field with name _192_168_1_1_ instead of _cs1Label_. Since the IP address in the log messages is changing frequently, a number of fields in the Elasticsearch index mapping quickly reaches the default limit of 1000 fields per index.
At that point, Graylog server logs messages such as this:
`message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_
exception, reason=Limit of total fields [1000] has been exceeded]]]`
## Expected Behavior
The key-value pair with mixed case letters in the field name such as _cs1Label=192.168.1.1_ should produce field name _cs1Label_ instead of using the value as a field name.
## Current Behavior
The key-value pair with mixed case letters in the field name such as _cs1Label=192.168.1.1_ produces a field name _192_168_1_1_ which is incorrect and has a potential of flooding the Elasticsearch index mapping with useless fields, preventing other fields of being automatically added to the index mapping.
## Possible Solution
This might be a bug in the CEF input code or the issue related to Elasticsearch behavior.
With all lowercase field names everything works as expected.
If there is no other way to fix the issue, it might be acceptable to put all field names to lowercase before processing them.
## Steps to Reproduce (for bugs)
1. Create CEF TCP input for testing purposes
2. Create index set for testing purposes
3. Create a test stream and make sure it gets log messages from the test CEF TCP input and stores them to the test index set created earlier
4. Send a test message to the test CEF TCP input. Message could look like this (the important part is the field named cs1Label):
`echo 'Nov 14 20:22:57 hostname CEF:0|DeviceVendor|DeviceProduct|1.0|713|TCP Connection Abort|3|cat=512 gcat=6 cs1Label=192.168.1.1 spt=51713 dst=192.168.2.2 dpt=443 msg="TCP connection abort received; TCP connection dropped" fw_action="drop"' | nc `
5. Check the index mapping for existence of the field _192_168_1_1_ using Elasticsearch API:
`curl -s -XGET "http://localhost:9200//_mapping?pretty"`
## Context
I was trying to send CEF formatted log messages from firewall to Graylog CEF TCP input.
I solved the problem using rsyslog in front of the Graylog.
I wrote a rsyslog template which puts the whole message to lowercase.
The template looks like this:
`template(name="cef" type="string" string="%timereported% %HOSTNAME% %syslogtag%%msg:::lowercase,drop-last-lf%\n")`
## Your Environment
* Graylog Version: 4.3.5-1, 4.3.9-1
* Java Version: openjdk version "11.0.16" 2022-07-19
* Elasticsearch Version: 7.10.2
* MongoDB Version: 4.2.22, 4.2.23
* Operating System: Debian 10.x
* Browser version: Irrelevant in this case
Contributor guide
Assessment
This issue has not been assessed yet.