elastic / elastic/logstash

Filter plugin/mutation to clean up empty values from object

Open
#15,403 0 comments 0 reactions 0 assignees View on GitHub
enhancement status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

It would be great to have a built-in feature that cleans up empty elements from a object, as its quite a common usecase when building pipelines.

In many cases, vendors sends the same schema each time, however that means when a value is empty/null, it will for example send an empty string or something like "N/A".

It would be great to have a mutation or filter plugin that handles it for you, that traverses the object recursively, and even supports arrays, in which you can define the object path where you want to traverse from, and any values that could be considered empty (default to empty string?).
Being able to provide a list of values that is considered empty could be great, though more an optional quality of life thing.

An example written that did not support arrays:

```
ruby {
init => '
def remove_empty_strings(obj, event, key_so_far="")
obj.each do |key, value|
actual_key = key_so_far + "[#{key}]"
if value.is_a?(String) && value.empty?
event.remove(actual_key)
elsif value.is_a?(Hash)
remove_empty_strings(value, event, actual_key)
end
end
end
'
code => '
remove_empty_strings(event.to_hash, event)
'
}
```

Another much better example written by Ry: https://gist.github.com/yaauie/8bbdcce4799248dd31f569e09d1ccb5f

Contributor guide

Open the contributing guide

Research direction

No repository file or test is named. Start by reviewing the requested built-in filter or mutation behavior and the linked Ruby examples, then define how recursive traversal, arrays, object paths, and configurable empty values should work. Done means the feature handles those cases consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, ruby
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.