Implement more efficient oneof in-memory structure
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 11
- Forks
- 6
- Avg merge
- 55m
- Merged PRs (30d)
- 1
Description
Sorted convertors from pdata keep the data points in memory. This results in large number of PointValue oneof structs in-mem. oneof is currently implemented as a plain struct instead of an aliased union, which results in large memory consumption and slowdown in operation with the increase in number of oneof choices. This is for example what we observe when we add ExpHistogramValue to PointValue (without even having a single data point with ExpHistogramValue present in the dataset):
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
SerializeNative/STEF/none-10 3.752m ± 2% 4.356m ± 5% +16.12% (p=0.000 n=10)
DeserializeNative/STEF/none-10 1.710m ± 0% 1.698m ± 0% -0.68% (p=0.003 n=10)
SerializeFromPdata/STEF/none-10 87.02m ± 1% 90.25m ± 1% +3.72% (p=0.001 n=10)
DeserializeToPdata/STEF/none-10 22.16m ± 1% 22.03m ± 1% ~ (p=0.393 n=10)
geomean 10.55m 11.01m +4.43%
│ old.txt │ new.txt │
│ sec/point │ sec/point vs base │
SerializeNative/STEF/none-10 56.12n ± 2% 65.16n ± 5% +16.12% (p=0.000 n=10)
DeserializeNative/STEF/none-10 25.57n ± 0% 25.41n ± 0% -0.66% (p=0.002 n=10)
SerializeFromPdata/STEF/none-10 1.302µ ± 1% 1.349µ ± 1% +3.69% (p=0.001 n=10)
DeserializeToPdata/STEF/none-10 331.4n ± 1% 329.4n ± 1% ~ (p=0.382 n=10)
geomean 157.7n 164.7n +4.42%
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
SerializeNative/STEF/none-10 3.521Mi ± 0% 3.515Mi ± 0% -0.19% (p=0.000 n=10)
DeserializeNative/STEF/none-10 836.2Ki ± 0% 839.4Ki ± 0% +0.38% (p=0.000 n=10)
SerializeFromPdata/STEF/none-10 124.6Mi ± 0% 141.2Mi ± 0% +13.34% (p=0.000 n=10)
DeserializeToPdata/STEF/none-10 29.81Mi ± 0% 29.81Mi ± 0% +0.01% (p=0.000 n=10)
geomean 10.17Mi 10.50Mi +3.23%
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
SerializeNative/STEF/none-10 2.773k ± 0% 2.802k ± 0% +1.05% (p=0.000 n=10)
DeserializeNative/STEF/none-10 1.230k ± 0% 1.261k ± 0% +2.52% (p=0.000 n=10)
SerializeFromPdata/STEF/none-10 256.3k ± 0% 256.3k ± 0% ~ (p=0.100 n=10)
DeserializeToPdata/STEF/none-10 623.3k ± 0% 623.3k ± 0% +0.00% (p=0.000 n=10)
geomean 27.17k 27.41k +0.89%
I confirmed that increase in serilization time is purely because of the increase of the size of the PointValue in-mem structure.
See if we can get inspiration from https://github.com/tigrannajaryan/govariant
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the PointValue oneof representation and the SerializeNative, SerializeFromPdata, and related benchmarks shown in the issue. Compare possible approaches with govariant, then verify that the structure uses less memory without regressing serialization performance or behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100