citusdata / citusdata/citus

Explore using Succinct Data Store for querying on compressed data

Open
#5,311 2 comments 0 reactions 0 assignees View on GitHub
columnar research
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

I saw your release announcement on 10.2 (congrats 🎉) and I was wondering if querying indexes always required decompressing at least one chunk.
It turns out there's a way that can query on compressed data.
The abstract of "Succinct: Enabling Queries on Compressed Data" looks very promising:
> Succinct is a data store that enables efficient queries directly
on a compressed representation of the input data.
Succinct uses a compression technique that allows random
access into the input, thus enabling efficient storage
and retrieval of data. In addition, Succinct natively
supports a wide range of queries including count and
search of arbitrary strings, range and wildcard queries.
What differentiates Succinct from previous techniques
is that Succinct supports these queries without storing
indexes — all the required information is embedded
within the compressed representation.
Evaluation on real-world datasets show that Succinct
requires an order of magnitude lower memory than systems
with similar functionality. Succinct thus pushes
more data in memory, and provides low query latency
for a larger range of input sizes than existing systems.

The [paper can be found here](https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/agarwal).

The Java [reference implementation](https://github.com/amplab/succinct/blob/master/core/README.md) is in Github, of course.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.