elastic / elastic/logstash

Codec Replacement Plan

Open
#5,124 2 comments 0 reactions 1 assignee Claimed by @andrewvc View on GitHub
design enhancement
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Logstash currently uses codecs. The codec abstraction externally makes (some) sense
but has a number of problems and limitations.
1. Codecs often have the dual responsibility of 1. splitting streams into discrete
events, and 2.) deserializing data.
2. Many codecs only decode or encode, not both. The interface doesn't let a plugin
declare that it only does one or the other.
3. Now that [event mills](https://github.com/elastic/logstash/issues/4858) exist the
tokenization of events happens before the codec stage.
4. Inputs and outputs waste time in the codec stage (they are often single threaded)
when this work could be done by the pipeline with greater efficiency.

Moving forward it makes more sense to take each codec and split it up into a filter
and a new Serializer object that encodes data. So, the JSON codec would become:
1. logstash-filter-json
2. logstash-serializer-json

We could keep the existing syntax but have it work with these new internals. The execution model would move from

```
# Current design
input(internal codec) -> queue -> filters -> output(internal codec)
# New design
input -> mill -> queue -> filters -> serializer -> output
```

To keep compatibility with the current config syntax the 'codec' directive would
do the following:
1. For inputs it would tag events coming out of a given input with a special directive
asking that a special filter be applied to that event before the normal filter chain.
2. For outputs all events going to an output would be run through the correct serializer first.
The output would need to implement a new interface as follows:

``` ruby
class MyOutput < LogStash::Output::Base
def receive_serialized(events_and_data)
events_and_data.each do |event, data|
# do something with event and data
end
end
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.