elastic / elastic/logstash

flat internal Event representation

Open
#1,968 5 comments 0 reactions 0 assignees View on GitHub
design discuss
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

an idea worth exploring is to move from an internal `Hash`/hierarchical event representation to a simpler flat representation. this is really some early brainstorming, please contribute ideas, thoughts, comments.

Overall goals: The LogStash::Event object supports nested structures and a special syntax for accessing nested field (`field references`). Internally, the object's JSON representation is basically the same as the object itself. Can be a hash of hash of hash, or whatever. From a memory usage perspective, this can consume lots of object references. From a serialization point of view, visiting all the objects can be costly. **We are interested in exploring some internal-representation improvements that should improve per-event memory usage and event serialization costs.**

basically instead of having an internal object hierarchy representation like

``` ruby
{
"message" => "foo",
"geoip" => {
"coords" => {
"latitude" => 45.5,
"longitude" => 73.5667
}
}
}
```

we could have something like

``` ruby
{
"message" => "foo",
"geoip.coords.latitude" => 45.5,
"geoip.coords.longitude" => 73.5667,
}
```

or, using the logstash path convention:

``` ruby
{
"[message]" => "foo",
"[geoip][coords][latitude]" => 45.5,
"[geoip][coords][longitude]" => 73.5667,
}
```

This could be done while preserving the current `Event` api, making this backward compatible.

Pros:
- we could get rid of the whole `Accessors` class which caches fields path to inner objects values. this would become essentially a 1:1 lookup. it would speed up lookups and remove the `Accessors` complexity.
- event serialization for persistence would become simpler and certainly faster

Cons:
- json input codec would need to change to use the Jackson streaming api and "convert" json object to a flat representation.
- same idea for json output codec, we'd need to produce a json object from a flat representation.
- we'd need to verify all usage of `Event#to_hash` and probably have to perform flat -> hierarchical conversion to create a proper Hash representation of the `Event`.

Thoughts?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.