Graylog2 / Graylog2/graylog2-server

Extract field from json_extracted field not working

Open
#4,229 18 comments 0 reactions 0 assignees View on GitHub
#S bug processing triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

I have 2 extractors.
1. extract json
```

{
"title": "json_extract",
"extractor_type": "json",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "",
"extractor_config": {
"flatten": false,
"list_separator": ", ",
"kv_separator": "=",
"key_prefix": "xxx.",
"key_separator": ".",
"replace_key_whitespace": false,
"key_whitespace_replacement": "_"
},
"condition_type": "none",
"condition_value": ""
},

```
send (lower) in order:
```

{
"title": "split xxx_datetime_date as store as timestamp_xxx",
"extractor_type": "split_and_index",
"converters": [
{
"type": "flexdate",
"config": {
"time_zone": "Poland"
}
}
],
"order": 2,
"cursor_strategy": "copy",
"source_field": "xxx_datetime_date",
"target_field": "timestamp_xxx",
"extractor_config": {
"index": 1,
"split_by": "."
},
"condition_type": "string",
"condition_value": "."
}
```

part of the json extract:
{...
"datetime":{"date":"2017-10-10 09:39:01.607127","timezone_type":3,"timezone":"Europe/Warsaw"}
...}

The fields are extracted to:
xxx_datetime_date: 2017-10-10 09:39:01.607127
xxx_timezone_type: 3
xxx_timezone: Europe/Warsaw

## Expected Behavior
Json Extractor works fine. But When I try to extract again field xxx_datetime_date to timestamp_xxx
or just copy the field to anoter filed. It's not working.

## Current Behavior
Second extractor on extracted json field (xxx_datetime_date) does not work.
No additional field are added to message.

Pipelines also not working.
```
rule "Appserver Parsing - Timestamp"
when
has_field("xxx_datetime_date")
then
let new_timestamp = parse_date(to_string($message.xxx_datetime_date), "yyyy-MM-dd HH:mm:ss.SSS");
set_field("xxx_pipline_timestamp", new_timestamp);
// If the timestamp is correct, rename the field
end
```

## Context
I cannot set timestamp from messages taht come as full json.

* Graylog Version: 2.3.0
* Elasticsearch Version: /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.4.4.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch
* MongoDB Version: v2.4.10
* Operating System: Ubuntu 16.10
* Browser version: Firefox 57 b5

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.