grafana / grafana/pyroscope

Respect physical data placement in parquet iterators

Open
#4,219 1 comment 0 reactions 1 assignee Claimed by @simonswine View on GitHub
storage
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

`parquet.List` of structs colocates fields on the same page. This means that we should never create an individual iterator for each of the columns in such cases ([example](https://github.com/grafana/pyroscope/blob/main/pkg/phlaredb/sample_merge.go#L120)): in fact, we fetch same pages repeatedly.

In addition, parquet reader issues a read operation for every ReadBufferPage size (+ page/group bounds), which prevents efficient streaming of data ranges from object storage.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.