Implemented a memory-compressed storage
- Dominant language
- C++
- Stars
- 334
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Sophisticated analyses use high-dimensional histograms, both disc and memory usage can become an issue. Disc usage is easy to fix by writing a compressed stream with Boost.Serialization and Boost.IOStreams. But this does not fix the memory issue.
It would be interesting to add an experimental memory-compressed storage and see how that performs. In-memory compression can be achieved in several ways.
* Zero-suppression: Some high-dimensional histograms contain a lot of zeros. A simple compression would just replace all the zeros with a code that indicates how many zeros are in the omitted gap. This is comparably easy to implement. Frequent re-allocations of memory in the beginning will be an issue, but that could be optimized over time.
* We could use an in-memory compressor like Blosc: https://www.blosc.org/pages/blosc-in-depth
Contributor guide
Assessment
This issue has not been assessed yet.