Explore using Succinct Data Store for querying on compressed data
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
I saw your release announcement on 10.2 (congrats 🎉) and I was wondering if querying indexes always required decompressing at least one chunk.
It turns out there's a way that can query on compressed data.
The abstract of "Succinct: Enabling Queries on Compressed Data" looks very promising:
> Succinct is a data store that enables efficient queries directly
on a compressed representation of the input data.
Succinct uses a compression technique that allows random
access into the input, thus enabling efficient storage
and retrieval of data. In addition, Succinct natively
supports a wide range of queries including count and
search of arbitrary strings, range and wildcard queries.
What differentiates Succinct from previous techniques
is that Succinct supports these queries without storing
indexes — all the required information is embedded
within the compressed representation.
Evaluation on real-world datasets show that Succinct
requires an order of magnitude lower memory than systems
with similar functionality. Succinct thus pushes
more data in memory, and provides low query latency
for a larger range of input sizes than existing systems.
The [paper can be found here](https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/agarwal).
The Java [reference implementation](https://github.com/amplab/succinct/blob/master/core/README.md) is in Github, of course.
Contributor guide
Assessment
This issue has not been assessed yet.