elastic / elastic/logstash

`@codec.decode` should support defining metadata.

Open
#4,289 9 comments 0 reactions 0 assignees View on GitHub
discuss
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

I have encountered an issue with the lumberjack and the beats plugin combined with the multiline codec and its really hard to fix without changing the API. The problem is related to spooling or the buffering of events within the codec code.

Some protocol support adding useful metadata (lumberjack, rabbitmq) to an event, but the `decode` API only work with string. It look like an async api but its more like a sync api.

See the following example:

``` ruby
raw_string = "bonjour la famille"
data = { "host" => "sashimi", "line" => raw_string }
@codec.decode(data) do |event|
event["host"] = data["host"]
end
```

This code is simple and its always right if the codec doesn't do buffering, since the proc will capture the data hash.

If the codec do some kind of buffering like the multiline filter it will only be right if new lines trigger a flush, in any other context like time based eviction or when we manually trigger a flush before shutting down the plugin that context will be lost.

We need revisit that api to add the possibility to add context/or metadata to the decode, or provide an object that can receive callbacks. Until then I have implemented the eviction block to only decorate and push to the queue the events, but the metadata wont be added to them :(

This issue is related to @guyboertje proposal for changing the interaction between the codec and the queues.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.