elastic / elastic/logstash

Create new filter plugin: logstash-filter-excerpt

Open
#4,452 2 comments 0 reactions 0 assignees View on GitHub
new plugin
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

There are situations where users process large events through Logstash and index it on to Elasticsearch. Typically you don't need the entire event indexed and searchable, but you'd still need to store the event for compliancy. It would be nice to have a filter that clones a big event, extracts an excerpt from it and indexes this to ES. Naively, an extraction can be the first 100 chars, or we can get fancier with it. Maybe even logstash-filter-summarization plugin based on [automatic summarization](https://en.wikipedia.org/wiki/Automatic_summarization). The original, large event can be routed to a long term storage like S3 or HDFS.

You can still achieve this today by tagging (based on size), cloning and using a ruby filter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.