influxdata / influxdata/influxdb

Implement Bonobo algorithm for influxdb, enabling 5 times better data compression

Open
#23,081 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__
Improve the compression algorithm like the Bonobo timeseries data compression algorithm sugggest here https://github.com/JsBergbau/Bonobo-Timeseries-Data-Compression
__Current behavior:__
Due to the nature of floating point numbers in binary system and the data from the decimal system there is often a lot of change in data, requiring unnecessary disk space. Measurements showed that compression can be improved by a factor of 5.
__Desired behavior:__
Bonobo timeseries data compression as explained in https://github.com/JsBergbau/Bonobo-Timeseries-Data-Compression is implemented.

__Alternatives considered:__
Currently I don't know and better solution to gain so much compression improvement.

__Use case:__
Why is this important (helps with prioritizing requests)?
Saving data by the factor of 5 is a lot of improvement, especially for 32 bit systems where database size is quite limited, see https://github.com/influxdata/influxdb/pull/12362

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked Bonobo Timeseries Data Compression project and the referenced InfluxDB pull request. Identify the relevant InfluxDB storage and compression entry points, then determine how the proposed algorithm could be integrated and benchmarked. Done means the algorithm is implemented and the claimed compression improvement is measured.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.