Graylog2 / Graylog2/graylog2-server
select_jsonpath() can't escape .
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
I'm trying to use *only Pipelines* to process logs from Zeek (Bro IDS) and has hit a dead end. The JSON log has key pairs that have dot "." in their name. The JSON log I get in $message.message is as following
```
{
"ts": "2019-03-27T23:21:39.126902Z",
"uid": "CKzRBx1OsfFfclbL1i",
"id.orig_h": "192.168.70.140",
"id.orig_p": 54274,
"id.resp_h": "216.58.200.100",
"id.resp_p": 443,
"proto": "tcp",
"conn_state": "OTH",
"local_orig": true,
"local_resp": false,
"missed_bytes": 0,
"history": "C",
"orig_pkts": 0,
"orig_ip_bytes": 0,
"resp_pkts": 0,
"resp_ip_bytes": 0,
"orig_l2_addr": "00:0c:29:0d:b2:c3",
"resp_l2_addr": "00:50:56:f5:03:c8"
}
```
I'm parsing that JSON blob with parse_json() and then picking each json object individually and placing into a new field. The nested JSON objects that have another . in them as `id.orig_h` will fail and won't work.
```
rule "parse conn.log"
when
$message.nsm_log == "zeek_conn"
then
let tmp=parse_json(to_string($message.message));
set_fields(select_jsonpath(tmp, {conn_uid:"$.uid"}));
set_fields(select_jsonpath(tmp, {protocol:"$.proto"}));;
set_fields(select_jsonpath(tmp, {src_ip:"$.id.orig_h"}));
set_fields(select_jsonpath(tmp, {src_port:"$.id.orig_p"}));
set_fields(select_jsonpath(tmp, {dest_ip:"$.id.resp_h"}));
set_fields(select_jsonpath(tmp, {dest_port:"$.id.resp_p"}));
end
```
## Expected Behavior
I should be able to pick the object `id.orig_h` by escaping the . character.
## Current Behavior
The particular set_fields() having such objects will just not work and those fields won't appear at all. Rest set_fields() works as expected.
## Possible Solution
Enable select_jsonselect() to deal with such object having dot "." in their names by escaping it.
## Steps to Reproduce (for bugs)
1. Ingest JSON formatted zeek logs
2. Use the above pipeline rules to tokenize the JSON blob in $message.message into individual fields.
## Context
Zeek is a network security monitoring application which generates metadata logs on network connections.
## Your Environment
* Graylog Version: 2.5.2+4f6d123
* Elasticsearch Version: elasticsearch-5.6.16-1
* MongoDB Version: mongodb-server-2.6.12-6.el7.x86_64
* Operating System: CentOS 7.6
* Browser version: Google Chrome 73.0.3683.86
Contributor guide
Assessment
This issue has not been assessed yet.