Columnar storage and small stripes
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
I am evaluating Citus Columnar to replace our Sql Server Clustered Columnstore indexes and I came upon stripes which are essentially the equivalent to segment in Sql Server. Based on the documentation, stripes needs to be as much large as possible (default 150,000) considering the fact how columnstore engines work to offer max performance. Our use case involves real time analytics (every 1 minute we do inserts) but we see an issue the way columnar works in Citus. Every minute we have between 5-1000 new rows. That creates a problem because based on the Citus documentation, tiny stripes will mess table.
In Sql Server this is avoided with the use of 'Delta rowgroup' which essentially works as a `buffer` until the number of rows reach 1,048,576 (number of rows on each segment) and then they are added and sealed in a segment.
Is this functionality something that is planned for Citus Columnar?
Contributor guide
Assessment
This issue has not been assessed yet.