apache / apache/arrow-julia

Reading only a subset of columns

Open
#78 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Please correct me if this is possible already. I looked through the source code and the documentation and did not find a clear way to do this: basically, I want to read a FeatherV2 file, but not mmap every single column. I already know which columns I need and I'd like to tell `Arrow.Table` the subset of columns I want read into memory.

This is similar to [this issue on Feather.jl](https://github.com/JuliaData/Feather.jl/issues/36).

This seems to be possible in the R arrow package using `col_select`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Arrow.Table entry point for reading FeatherV2 files and review how column selection and memory mapping are handled. Compare the requested behavior with Feather.jl issue 36 and R arrow's col_select option; done means callers can select columns without mapping every column into memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.