Graylog2 / Graylog2/graylog2-server

select_jsonpath() can't escape .

Open
#5,815 1 comment 0 reactions 0 assignees View on GitHub
processing to-verify
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

I'm trying to use *only Pipelines* to process logs from Zeek (Bro IDS) and has hit a dead end. The JSON log has key pairs that have dot "." in their name. The JSON log I get in $message.message is as following

```
{
"ts": "2019-03-27T23:21:39.126902Z",
"uid": "CKzRBx1OsfFfclbL1i",
"id.orig_h": "192.168.70.140",
"id.orig_p": 54274,
"id.resp_h": "216.58.200.100",
"id.resp_p": 443,
"proto": "tcp",
"conn_state": "OTH",
"local_orig": true,
"local_resp": false,
"missed_bytes": 0,
"history": "C",
"orig_pkts": 0,
"orig_ip_bytes": 0,
"resp_pkts": 0,
"resp_ip_bytes": 0,
"orig_l2_addr": "00:0c:29:0d:b2:c3",
"resp_l2_addr": "00:50:56:f5:03:c8"
}
```

I'm parsing that JSON blob with parse_json() and then picking each json object individually and placing into a new field. The nested JSON objects that have another . in them as `id.orig_h` will fail and won't work.

```
rule "parse conn.log"
when
$message.nsm_log == "zeek_conn"
then
let tmp=parse_json(to_string($message.message));
set_fields(select_jsonpath(tmp, {conn_uid:"$.uid"}));
set_fields(select_jsonpath(tmp, {protocol:"$.proto"}));;
set_fields(select_jsonpath(tmp, {src_ip:"$.id.orig_h"}));
set_fields(select_jsonpath(tmp, {src_port:"$.id.orig_p"}));
set_fields(select_jsonpath(tmp, {dest_ip:"$.id.resp_h"}));
set_fields(select_jsonpath(tmp, {dest_port:"$.id.resp_p"}));
end
```

## Expected Behavior

I should be able to pick the object `id.orig_h` by escaping the . character.

## Current Behavior

The particular set_fields() having such objects will just not work and those fields won't appear at all. Rest set_fields() works as expected.

## Possible Solution

Enable select_jsonselect() to deal with such object having dot "." in their names by escaping it.

## Steps to Reproduce (for bugs)

1. Ingest JSON formatted zeek logs
2. Use the above pipeline rules to tokenize the JSON blob in $message.message into individual fields.

## Context

Zeek is a network security monitoring application which generates metadata logs on network connections.

## Your Environment

* Graylog Version: 2.5.2+4f6d123
* Elasticsearch Version: elasticsearch-5.6.16-1
* MongoDB Version: mongodb-server-2.6.12-6.el7.x86_64
* Operating System: CentOS 7.6
* Browser version: Google Chrome 73.0.3683.86

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.