Filter plugin/mutation to clean up empty values from object
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
It would be great to have a built-in feature that cleans up empty elements from a object, as its quite a common usecase when building pipelines.
In many cases, vendors sends the same schema each time, however that means when a value is empty/null, it will for example send an empty string or something like "N/A".
It would be great to have a mutation or filter plugin that handles it for you, that traverses the object recursively, and even supports arrays, in which you can define the object path where you want to traverse from, and any values that could be considered empty (default to empty string?).
Being able to provide a list of values that is considered empty could be great, though more an optional quality of life thing.
An example written that did not support arrays:
```
ruby {
init => '
def remove_empty_strings(obj, event, key_so_far="")
obj.each do |key, value|
actual_key = key_so_far + "[#{key}]"
if value.is_a?(String) && value.empty?
event.remove(actual_key)
elsif value.is_a?(Hash)
remove_empty_strings(value, event, actual_key)
end
end
end
'
code => '
remove_empty_strings(event.to_hash, event)
'
}
```
Another much better example written by Ry: https://gist.github.com/yaauie/8bbdcce4799248dd31f569e09d1ccb5f
Contributor guide
Research direction
No repository file or test is named. Start by reviewing the requested built-in filter or mutation behavior and the linked Ruby examples, then define how recursive traversal, arrays, object paths, and configurable empty values should work. Done means the feature handles those cases consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, ruby
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100