flat internal Event representation
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
an idea worth exploring is to move from an internal `Hash`/hierarchical event representation to a simpler flat representation. this is really some early brainstorming, please contribute ideas, thoughts, comments.
Overall goals: The LogStash::Event object supports nested structures and a special syntax for accessing nested field (`field references`). Internally, the object's JSON representation is basically the same as the object itself. Can be a hash of hash of hash, or whatever. From a memory usage perspective, this can consume lots of object references. From a serialization point of view, visiting all the objects can be costly. **We are interested in exploring some internal-representation improvements that should improve per-event memory usage and event serialization costs.**
basically instead of having an internal object hierarchy representation like
``` ruby
{
"message" => "foo",
"geoip" => {
"coords" => {
"latitude" => 45.5,
"longitude" => 73.5667
}
}
}
```
we could have something like
``` ruby
{
"message" => "foo",
"geoip.coords.latitude" => 45.5,
"geoip.coords.longitude" => 73.5667,
}
```
or, using the logstash path convention:
``` ruby
{
"[message]" => "foo",
"[geoip][coords][latitude]" => 45.5,
"[geoip][coords][longitude]" => 73.5667,
}
```
This could be done while preserving the current `Event` api, making this backward compatible.
Pros:
- we could get rid of the whole `Accessors` class which caches fields path to inner objects values. this would become essentially a 1:1 lookup. it would speed up lookups and remove the `Accessors` complexity.
- event serialization for persistence would become simpler and certainly faster
Cons:
- json input codec would need to change to use the Jackson streaming api and "convert" json object to a flat representation.
- same idea for json output codec, we'd need to produce a json object from a flat representation.
- we'd need to verify all usage of `Event#to_hash` and probably have to perform flat -> hierarchical conversion to create a proper Hash representation of the `Event`.
Thoughts?
Contributor guide
Assessment
This issue has not been assessed yet.