apache / apache/arrow

[C++][Parquet] Reuse memory of passed array in ConvertToDecimal

Open
#38,881 1 comment 6 reactions 1 assignee Claimed by @Hattonuri View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

I am reading an arrow::ListArray of Decimal128 from FixedLengthBinaryArray[16] in parquets and see on flamegraph that RawBytesToDecimalBytes consumes large amount of time and mostly due to page faults.
image
image

I assume that the problem happens here
https://github.com/apache/arrow/blob/eb5de184a7e5d02f98526332ace54250417bd232/cpp/src/parquet/arrow/reader_internal.cc#L559-L576
when arrow allocates new buffer for decimals and then parses FLBA into it.
As FLBA type has the same amount of memory per element i guess that this function should reuse given array data.

Possible easy(but very dirty) solution could be static_cast given array to Decimal array (as it does not have additional fields - it wouldn't be an error), const_cast raw_values of it and reverse byte order in-place
Another benefit is reducing memory usage of reading them

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.