apache / apache/arrow-go

[Go][Parquet] Looking for Memory-friendly way to seek & extract data from parquet columns

Open
#38 1 comment 0 reactions 0 assignees View on GitHub
Type: usage
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the usage question you have. Please include as many useful details as possible.

Hi,

beeing new to Apache Arrow I'm a little confused about the different options to interact with Parquet files. The documentation in the Go library is in many places very sparse and existing examples from various sources don't seem to match my use case.

**The question is:**
Given that you have a parquet file containing serveral thousand rows each with an _ID_ column and a _Data_ column, where the data column holds some larger blob, how do you seek certain rows based on their ID column and extract the data of the _Data_ column in an efficient and memory-friendly way?

By 'memory-friendly' I mean that only the relevant values should be read from the parquet files and loaded into memory, not the whole column, rowgroup, batch or chunk. Reading the ID column completely into memory would be fine, but not the blob data.

I tried the variant with creating a pqarrow.RecordReader based on a pqarrow.FileReader based on parquet.Reader, but it seems that the Record batches always load the the whole batch (incl. all column data) into memory, not just when loading the value of of a column entry by index. While this approach works as desired, it has a very high memory usage due to the large blobs.

I also tried to extract the relevant row-indexes in a first sweep to then somehow only retrieve these rows from the Data column in a second sweep, but I could not find a way that improved the first approach.

There is probably a simple way (without using pqarrow?) by just iterating over the parquet file rowgroups, but the usage of the available datastructures are not reallly documented well (FieldReaders, chunks, etc.)

Btw, doing the same thing with DuckDb works very well and is noticably lighter on memory than the RecordReader approach, but including that library for the simple seek&extract use case is somewhat overkill and I would prefer to avoid it.

Thanks for any hints

Jochen Mehlhorn , Mercedes-Benz Tech Innovation GmbH

[Provider Information](https://github.com/Daimler/daimler-foss/blob/master/PROVIDER_INFORMATION.md)

### Component(s)

Go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.