Codec Replacement Plan
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Logstash currently uses codecs. The codec abstraction externally makes (some) sense
but has a number of problems and limitations.
1. Codecs often have the dual responsibility of 1. splitting streams into discrete
events, and 2.) deserializing data.
2. Many codecs only decode or encode, not both. The interface doesn't let a plugin
declare that it only does one or the other.
3. Now that [event mills](https://github.com/elastic/logstash/issues/4858) exist the
tokenization of events happens before the codec stage.
4. Inputs and outputs waste time in the codec stage (they are often single threaded)
when this work could be done by the pipeline with greater efficiency.
Moving forward it makes more sense to take each codec and split it up into a filter
and a new Serializer object that encodes data. So, the JSON codec would become:
1. logstash-filter-json
2. logstash-serializer-json
We could keep the existing syntax but have it work with these new internals. The execution model would move from
```
# Current design
input(internal codec) -> queue -> filters -> output(internal codec)
# New design
input -> mill -> queue -> filters -> serializer -> output
```
To keep compatibility with the current config syntax the 'codec' directive would
do the following:
1. For inputs it would tag events coming out of a given input with a special directive
asking that a special filter be applied to that event before the normal filter chain.
2. For outputs all events going to an output would be run through the correct serializer first.
The output would need to implement a new interface as follows:
``` ruby
class MyOutput < LogStash::Output::Base
def receive_serialized(events_and_data)
events_and_data.each do |event, data|
# do something with event and data
end
end
```
Contributor guide
Assessment
This issue has not been assessed yet.