Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, _unless_ I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.
Here are two small config files that I can use to recreate this issue:
**redis-send.conf**
```
input {
stdin {}
}
output {
redis {
data_type => "list"
key => "logs"
}
}
```
**redis-receive.conf**
```
input {
stdin {}
redis {
data_type => "list"
key => "logs"
}
}
filter {
csv {
columns => ["a", "b", "c", "d", "e"]
}
mutate {
remove_field => "a"
}
}
output {
stdout {
codec => "rubydebug"
}
}
```
On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:
```
{
"message" => [
[0] "1,2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:12:44.173Z",
"host" => "starscream",
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
```
Just like I expect.
However, leaving the first field empty:
```
{
"message" => [
[0] ",2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:17:54.859Z",
"host" => "starscream",
"a" => nil,
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
```
It does not remove the field "a".
But if I type the same string in stdin on the receiver agent, it removes the field:
```
{
"message" => [
[0] ",2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:18:24.907Z",
"host" => "starscream",
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
```
I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.
Contributor guide
Assessment
This issue has not been assessed yet.