grafana / grafana/pyroscope

Flushing RowGroups of profile based on size

Open
#2,756 0 comments 0 reactions 0 assignees View on GitHub
ops performance v1
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

So far we're been flushing Rowgroups every 100k profiles and we recently faced issues during compaction where parquet reader and writers would start using a lot of memory because of unbalanced rowgroup size.

In fact we're seen some Rowgroups ranging from 100MB to 3GB, this mostly because of the variable in amount of samples a single profile can contains which is greatly affected by the scrape/push interval, profile types or applications.

To make memory consumption predicable we have to make those rowgroup size predictable and this why I suggest we change how we decide to flush rowgroup in the ingester and compactors.

We should flush rowgroup once we reach a certain in-memory size, and we should aim for compressed/encoded rowgroups of ~512MB. Parquet suggest between 512 and 1GB.

I suggest we in fact run couple of performance test with difference rowgroup size.

In the future this will be very useful also for the query path to parallelized based on rowgroups.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.