boostorg / boostorg/histogram

Implemented a memory-compressed storage

Open
#318 1 comment 0 reactions 0 assignees View on GitHub
easy enhancement help wanted
Dominant language
C++
Stars
334
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Sophisticated analyses use high-dimensional histograms, both disc and memory usage can become an issue. Disc usage is easy to fix by writing a compressed stream with Boost.Serialization and Boost.IOStreams. But this does not fix the memory issue.

It would be interesting to add an experimental memory-compressed storage and see how that performs. In-memory compression can be achieved in several ways.

* Zero-suppression: Some high-dimensional histograms contain a lot of zeros. A simple compression would just replace all the zeros with a code that indicates how many zeros are in the omitted gap. This is comparably easy to implement. Frequent re-allocations of memory in the beginning will be an issue, but that could be optimized over time.
* We could use an in-memory compressor like Blosc: https://www.blosc.org/pages/blosc-in-depth

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.