splunk / splunk/stef

Implement more efficient oneof in-memory structure

Open
#72 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
11
Forks
6
Avg merge
55m
Merged PRs (30d)
1

Description

Sorted convertors from pdata keep the data points in memory. This results in large number of PointValue oneof structs in-mem. oneof is currently implemented as a plain struct instead of an aliased union, which results in large memory consumption and slowdown in operation with the increase in number of oneof choices. This is for example what we observe when we add ExpHistogramValue to PointValue (without even having a single data point with ExpHistogramValue present in the dataset):

                                │   old.txt   │               new.txt               │
                                │   sec/op    │   sec/op     vs base                │
SerializeNative/STEF/none-10      3.752m ± 2%   4.356m ± 5%  +16.12% (p=0.000 n=10)
DeserializeNative/STEF/none-10    1.710m ± 0%   1.698m ± 0%   -0.68% (p=0.003 n=10)
SerializeFromPdata/STEF/none-10   87.02m ± 1%   90.25m ± 1%   +3.72% (p=0.001 n=10)
DeserializeToPdata/STEF/none-10   22.16m ± 1%   22.03m ± 1%        ~ (p=0.393 n=10)
geomean                           10.55m        11.01m        +4.43%

                                │   old.txt   │               new.txt               │
                                │  sec/point  │  sec/point   vs base                │
SerializeNative/STEF/none-10      56.12n ± 2%   65.16n ± 5%  +16.12% (p=0.000 n=10)
DeserializeNative/STEF/none-10    25.57n ± 0%   25.41n ± 0%   -0.66% (p=0.002 n=10)
SerializeFromPdata/STEF/none-10   1.302µ ± 1%   1.349µ ± 1%   +3.69% (p=0.001 n=10)
DeserializeToPdata/STEF/none-10   331.4n ± 1%   329.4n ± 1%        ~ (p=0.382 n=10)
geomean                           157.7n        164.7n        +4.42%

                                │   old.txt    │               new.txt                │
                                │     B/op     │     B/op      vs base                │
SerializeNative/STEF/none-10      3.521Mi ± 0%   3.515Mi ± 0%   -0.19% (p=0.000 n=10)
DeserializeNative/STEF/none-10    836.2Ki ± 0%   839.4Ki ± 0%   +0.38% (p=0.000 n=10)
SerializeFromPdata/STEF/none-10   124.6Mi ± 0%   141.2Mi ± 0%  +13.34% (p=0.000 n=10)
DeserializeToPdata/STEF/none-10   29.81Mi ± 0%   29.81Mi ± 0%   +0.01% (p=0.000 n=10)
geomean                           10.17Mi        10.50Mi        +3.23%

                                │   old.txt   │              new.txt               │
                                │  allocs/op  │  allocs/op   vs base               │
SerializeNative/STEF/none-10      2.773k ± 0%   2.802k ± 0%  +1.05% (p=0.000 n=10)
DeserializeNative/STEF/none-10    1.230k ± 0%   1.261k ± 0%  +2.52% (p=0.000 n=10)
SerializeFromPdata/STEF/none-10   256.3k ± 0%   256.3k ± 0%       ~ (p=0.100 n=10)
DeserializeToPdata/STEF/none-10   623.3k ± 0%   623.3k ± 0%  +0.00% (p=0.000 n=10)
geomean                           27.17k        27.41k       +0.89%

I confirmed that increase in serilization time is purely because of the increase of the size of the PointValue in-mem structure.

See if we can get inspiration from https://github.com/tigrannajaryan/govariant

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the PointValue oneof representation and the SerializeNative, SerializeFromPdata, and related benchmarks shown in the issue. Compare possible approaches with govariant, then verify that the structure uses less memory without regressing serialization performance or behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.