elastic / elastic/logstash

logstash forgets data type

Open
#4,450 11 comments 0 reactions 1 assignee Claimed by @purbon View on GitHub
unconfirmed
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

I have two setups parsing sensu messages.

filebeat -> logstash1 -> gelf (Graylog)
filebeat -> logstash1 -> rabbitmq <- logstash2 -> gelf (Graylog)

In the second setup the data type of a float value is represented as "0.363E0" in Graylog. In the first setup it is stored as 0.363.

The logstash2 instance is quite simple there is no filtering at all just, input rabbitmq output gelf.

The logstash1 instance parses sensu messages lile this using the json filter

```
{"timestamp":"2016-01-11T16:47:35.185132+0000","level":"info","message":"publishing check result","payload":{"client":"i-e41b056d","check":{"name":"cpu_metrics","issued":1452530854,"command":"metrics-cpu.rb --scheme :::scheme_prefix::::::name:::.cpu","type":"metric","subscribers":["linux"],"handlers":["metrics"],"interval":60,"occurrences":2,"executed":1452530854,"duration":0.363,"output":"sensu.sensu.i-e41b056d.cpu.total.user 2819546 1452530855\nsensu.sensu.i-e41b056d.cpu.total.nice 6686 1452530855\nsensu.sensu.i-e41b056d.cpu.total.system 488912 1452530855\nsensu.sensu.i-e41b056d.cpu.total.idle 41970433 1452530855\nsensu.sensu.i-e41b056d.cpu.total.iowait 16862 1452530855\nsensu.sensu.i-e41b056d.cpu.total.irq 40 1452530855\nsensu.sensu.i-e41b056d.cpu.total.softirq 10503 1452530855\nsensu.sensu.i-e41b056d.cpu.total.steal 57530 1452530855\nsensu.sensu.i-e41b056d.cpu.total.guest 0 1452530855\nsensu.sensu.i-e41b056d.cpu.total. 0 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.user 2819546 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.nice 6686 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.system 488912 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.idle 41970433 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.iowait 16862 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.irq 40 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.softirq 10503 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.steal 57530 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.guest 0 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0. 0 1452530855\nsensu.sensu.i-e41b056d.cpu.intr 0 1452530855\nsensu.sensu.i-e41b056d.cpu.ctxt 125320448 1452530855\nsensu.sensu.i-e41b056d.cpu.btime 1452077217 1452530855\nsensu.sensu.i-e41b056d.cpu.processes 1102990 1452530855\nsensu.sensu.i-e41b056d.cpu.procs_running 2 1452530855\nsensu.sensu.i-e41b056d.cpu.procs_blocked 0 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu_count 1 1452530855\n\n","status":0}}}
```

My filter config looks like this

```
json {
source => message
target => "sensu"
add_tag => "sensu_found"
}

if ("sensu_found" in [tags]) {
mutate {
add_field => ["short_message", "%{[sensu][message]}"]
replace => { "severity" => "%{[sensu][level]}" }
remove_field => [ "[sensu][level]", "[sensu][message]" ]
}

#2015-12-11T14:05:07.154701+0000
date {
match => [ "[sensu][timestamp]", "ISO8601" ]
locale => "Locale.US"
add_tag => ["timestamp_changed"]
remove_field => [ "[sensu][timestamp]" ]
}

if([sensu][payload][check]) {
mutate {
rename => ["[sensu][payload][client]", "sensu_client"]
rename => ["[sensu][payload][check][duration]", "sensu_check_duration"]
rename => ["[sensu][payload][check][interval]", "sensu_check_interval"]
rename => ["[sensu][payload][check][name]", "sensu_check_name"]
rename => ["[sensu][payload][check][command]", "sensu_check_command"]
rename => ["[sensu][payload][check][type]", "sensu_check_type"]
rename => ["[sensu][payload][check][status]", "sensu_check_status"]
rename => ["[sensu][payload][check][output]", "sensu_check_output"]
rename => ["[sensu][payload][check][occurrences]", "sensu_check_occurrences"]
convert => [ "sensu_check_duration", "float" ]
convert => [ "sensu_check_interval", "integer" ]
remove_field => [ "[sensu][payload]" ]
}
}

if([sensu][event][check]) {
mutate {
convert => [ "[sensu][event][check][duration]", "float" ]
convert => [ "[sensu][event][check][interval]", "integer" ]
rename => ["[sensu][event][client][name]", "sensu_client"]
rename => ["[sensu][event][check][duration]", "sensu_check_duration"]
rename => ["[sensu][event][check][interval]", "sensu_check_interval"]
rename => ["[sensu][event][check][name]", "sensu_check_name"]
rename => ["[sensu][event][check][command]", "sensu_check_command"]
rename => ["[sensu][event][check][type]", "sensu_check_type"]
rename => ["[sensu][event][check][status]", "sensu_check_status"]
rename => ["[sensu][event][check][output]", "sensu_check_output"]
rename => ["[sensu][event][check][occurrences]", "sensu_check_occurrences"]
rename => ["[sensu][event][check][history]", "sensu_check_history"]
rename => ["[sensu][event][check][total_state_change]", "sensu_check_total_state_change"]
rename => ["[sensu][event][id]", "sensu_event_id"]
rename => ["[sensu][event][occurrences]", "sensu_event_occurrences"]
convert => [ "sensu_event_occurrences", "integer" ]
convert => [ "sensu_check_duration", "float" ]
convert => [ "sensu_check_interval", "integer" ]
rename => ["[sensu][event][action]", "sensu_event_action"]
remove_field => [ "[sensu][event]" ]
}
}
```

Why is the data type changed here?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.