logstash forgets data type
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
I have two setups parsing sensu messages.
filebeat -> logstash1 -> gelf (Graylog)
filebeat -> logstash1 -> rabbitmq <- logstash2 -> gelf (Graylog)
In the second setup the data type of a float value is represented as "0.363E0" in Graylog. In the first setup it is stored as 0.363.
The logstash2 instance is quite simple there is no filtering at all just, input rabbitmq output gelf.
The logstash1 instance parses sensu messages lile this using the json filter
```
{"timestamp":"2016-01-11T16:47:35.185132+0000","level":"info","message":"publishing check result","payload":{"client":"i-e41b056d","check":{"name":"cpu_metrics","issued":1452530854,"command":"metrics-cpu.rb --scheme :::scheme_prefix::::::name:::.cpu","type":"metric","subscribers":["linux"],"handlers":["metrics"],"interval":60,"occurrences":2,"executed":1452530854,"duration":0.363,"output":"sensu.sensu.i-e41b056d.cpu.total.user 2819546 1452530855\nsensu.sensu.i-e41b056d.cpu.total.nice 6686 1452530855\nsensu.sensu.i-e41b056d.cpu.total.system 488912 1452530855\nsensu.sensu.i-e41b056d.cpu.total.idle 41970433 1452530855\nsensu.sensu.i-e41b056d.cpu.total.iowait 16862 1452530855\nsensu.sensu.i-e41b056d.cpu.total.irq 40 1452530855\nsensu.sensu.i-e41b056d.cpu.total.softirq 10503 1452530855\nsensu.sensu.i-e41b056d.cpu.total.steal 57530 1452530855\nsensu.sensu.i-e41b056d.cpu.total.guest 0 1452530855\nsensu.sensu.i-e41b056d.cpu.total. 0 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.user 2819546 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.nice 6686 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.system 488912 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.idle 41970433 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.iowait 16862 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.irq 40 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.softirq 10503 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.steal 57530 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0.guest 0 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu0. 0 1452530855\nsensu.sensu.i-e41b056d.cpu.intr 0 1452530855\nsensu.sensu.i-e41b056d.cpu.ctxt 125320448 1452530855\nsensu.sensu.i-e41b056d.cpu.btime 1452077217 1452530855\nsensu.sensu.i-e41b056d.cpu.processes 1102990 1452530855\nsensu.sensu.i-e41b056d.cpu.procs_running 2 1452530855\nsensu.sensu.i-e41b056d.cpu.procs_blocked 0 1452530855\nsensu.sensu.i-e41b056d.cpu.cpu_count 1 1452530855\n\n","status":0}}}
```
My filter config looks like this
```
json {
source => message
target => "sensu"
add_tag => "sensu_found"
}
if ("sensu_found" in [tags]) {
mutate {
add_field => ["short_message", "%{[sensu][message]}"]
replace => { "severity" => "%{[sensu][level]}" }
remove_field => [ "[sensu][level]", "[sensu][message]" ]
}
#2015-12-11T14:05:07.154701+0000
date {
match => [ "[sensu][timestamp]", "ISO8601" ]
locale => "Locale.US"
add_tag => ["timestamp_changed"]
remove_field => [ "[sensu][timestamp]" ]
}
if([sensu][payload][check]) {
mutate {
rename => ["[sensu][payload][client]", "sensu_client"]
rename => ["[sensu][payload][check][duration]", "sensu_check_duration"]
rename => ["[sensu][payload][check][interval]", "sensu_check_interval"]
rename => ["[sensu][payload][check][name]", "sensu_check_name"]
rename => ["[sensu][payload][check][command]", "sensu_check_command"]
rename => ["[sensu][payload][check][type]", "sensu_check_type"]
rename => ["[sensu][payload][check][status]", "sensu_check_status"]
rename => ["[sensu][payload][check][output]", "sensu_check_output"]
rename => ["[sensu][payload][check][occurrences]", "sensu_check_occurrences"]
convert => [ "sensu_check_duration", "float" ]
convert => [ "sensu_check_interval", "integer" ]
remove_field => [ "[sensu][payload]" ]
}
}
if([sensu][event][check]) {
mutate {
convert => [ "[sensu][event][check][duration]", "float" ]
convert => [ "[sensu][event][check][interval]", "integer" ]
rename => ["[sensu][event][client][name]", "sensu_client"]
rename => ["[sensu][event][check][duration]", "sensu_check_duration"]
rename => ["[sensu][event][check][interval]", "sensu_check_interval"]
rename => ["[sensu][event][check][name]", "sensu_check_name"]
rename => ["[sensu][event][check][command]", "sensu_check_command"]
rename => ["[sensu][event][check][type]", "sensu_check_type"]
rename => ["[sensu][event][check][status]", "sensu_check_status"]
rename => ["[sensu][event][check][output]", "sensu_check_output"]
rename => ["[sensu][event][check][occurrences]", "sensu_check_occurrences"]
rename => ["[sensu][event][check][history]", "sensu_check_history"]
rename => ["[sensu][event][check][total_state_change]", "sensu_check_total_state_change"]
rename => ["[sensu][event][id]", "sensu_event_id"]
rename => ["[sensu][event][occurrences]", "sensu_event_occurrences"]
convert => [ "sensu_event_occurrences", "integer" ]
convert => [ "sensu_check_duration", "float" ]
convert => [ "sensu_check_interval", "integer" ]
rename => ["[sensu][event][action]", "sensu_event_action"]
remove_field => [ "[sensu][event]" ]
}
}
```
Why is the data type changed here?
Contributor guide
Assessment
This issue has not been assessed yet.