Flushing RowGroups of profile based on size
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
So far we're been flushing Rowgroups every 100k profiles and we recently faced issues during compaction where parquet reader and writers would start using a lot of memory because of unbalanced rowgroup size.
In fact we're seen some Rowgroups ranging from 100MB to 3GB, this mostly because of the variable in amount of samples a single profile can contains which is greatly affected by the scrape/push interval, profile types or applications.
To make memory consumption predicable we have to make those rowgroup size predictable and this why I suggest we change how we decide to flush rowgroup in the ingester and compactors.
We should flush rowgroup once we reach a certain in-memory size, and we should aim for compressed/encoded rowgroups of ~512MB. Parquet suggest between 512 and 1GB.
I suggest we in fact run couple of performance test with difference rowgroup size.
In the future this will be very useful also for the query path to parallelized based on rowgroups.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.