elastic / elastic/logstash

Logstash 1.5.0, Unable to remove field with nil value if it comes from redis input

Open
#3,310 1 comment 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Running logstash 1.5.0, I have a file with csv data that I read into logstash and parse using the csv plugin. Some fields will always be empty and gets a value of null. I then do some checking and remove these fields before shipping to elasticsearch. This all works fine, _unless_ I first ship the csv data to redis, and then parse it, then I'm unable to remove the fields with value null.

Here are two small config files that I can use to recreate this issue:

**redis-send.conf**

```
input {
stdin {}
}

output {
redis {
data_type => "list"
key => "logs"
}
}
```

**redis-receive.conf**

```
input {
stdin {}
redis {
data_type => "list"
key => "logs"
}
}

filter {
csv {
columns => ["a", "b", "c", "d", "e"]
}
mutate {
remove_field => "a"
}
}

output {
stdout {
codec => "rubydebug"
}
}

```

On the sender, I'm typing the string "1,2,3,4,5" to stdin. On the receiver agent, I get this output:

```

{
"message" => [
[0] "1,2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:12:44.173Z",
"host" => "starscream",
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}

```

Just like I expect.

However, leaving the first field empty:

```
{
"message" => [
[0] ",2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:17:54.859Z",
"host" => "starscream",
"a" => nil,
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}

```

It does not remove the field "a".

But if I type the same string in stdin on the receiver agent, it removes the field:

```
{
"message" => [
[0] ",2,3,4,5"
],
"@version" => "1",
"@timestamp" => "2015-05-22T14:18:24.907Z",
"host" => "starscream",
"b" => "2",
"c" => "3",
"d" => "4",
"e" => "5"
}
```

I've tried the above in logstash 1.4.2 as well but didn't encounter the issue in that version.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.