`@codec.decode` should support defining metadata.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
I have encountered an issue with the lumberjack and the beats plugin combined with the multiline codec and its really hard to fix without changing the API. The problem is related to spooling or the buffering of events within the codec code.
Some protocol support adding useful metadata (lumberjack, rabbitmq) to an event, but the `decode` API only work with string. It look like an async api but its more like a sync api.
See the following example:
``` ruby
raw_string = "bonjour la famille"
data = { "host" => "sashimi", "line" => raw_string }
@codec.decode(data) do |event|
event["host"] = data["host"]
end
```
This code is simple and its always right if the codec doesn't do buffering, since the proc will capture the data hash.
If the codec do some kind of buffering like the multiline filter it will only be right if new lines trigger a flush, in any other context like time based eviction or when we manually trigger a flush before shutting down the plugin that context will be lost.
We need revisit that api to add the possibility to add context/or metadata to the decode, or provide an object that can receive callbacks. Until then I have implemented the eviction block to only decorate and push to the queue the events, but the metadata wont be added to them :(
This issue is related to @guyboertje proposal for changing the interaction between the codec and the queues.
Contributor guide
Assessment
This issue has not been assessed yet.