apache / apache/arrow-julia

Feather file with compression and larger than RAM

Open
#340 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Last time I checked, `mmap` breaks down for files with compression. This is understandable because the compressed buffers clearly can't be re-interpreted without inflation.

But the larger the file is more likely to be compressed, can we decompress only a single "row group" (and only the relevant columns, of course) on the fly yet? -- this is for the case when a user is doing per-row iteration

if user access the table by `tbl[range, range]`, then clearly, we might need to read more than one row-group and chop off head/tails depending on where the overlap is

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how compressed Feather files are handled through mmap and how per-row iteration accesses a table. Then examine the tbl[range, range] path and the row-group and column boundaries described in the issue. Done means compressed files larger than RAM can decompress only the needed row groups and columns for iteration and ranged table access.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.