apache / apache/arrow-go

[Go][Parquet] Use custom allocator in ReaderProperties.GetStream

Open
#540 8 comments 1 reaction 0 assignees View on GitHub
Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the enhancement requested

We're testing reading parquet files from S3 and I'm looking at the memory profile. We're using a 2GB file for testing and in the profile I see that the `make` call in https://github.com/apache/arrow-go/blob/c8d24781add849d22169a27a8bd6d00a86fbdde1/parquet/reader_properties.go#L80 allocates about 2GB of data, which is about a third of the total allocations that our test that reads the whole file does.

I see that the ReaderProperties has an allocator field, but the allocation there is made using the standard Go call. Is there a reason why the custom allocator is not used there? I wanted to test using a custom allocator, so maybe we can switch the `make` call to `r.alloc.Allocate` ?

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.