apache / apache/arrow-julia

`Arrow.write` doesn't play nicely when the target table is lazy.

Open
#244 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

Imagine a table `mytable` ~ 10x RAM size, the `Tables.partitions` is naturally defined as a lazy iterator:
```julia
Tables.partitions(t::MyTable) = (t[range] for range in range_partition(t))
```

But when subsequently calling: `Arrow.write("out.arrow", mytable; ntasks=1)`, because the:
https://github.com/JuliaData/Arrow.jl/blob/967e3c67437219698afae889a13e881b60033fe3/src/write.jl#L127

the table is materialized in RAM anyway (because `t[range]` triggers materialization). Is there any blessed way to make this a truly sequential process such that we never hold more than one partition in RAM at any time?

related: https://github.com/tamasgal/UnROOT.jl/issues/114#issuecomment-925401278

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at src/write.jl around line 127 and trace Arrow.write("out.arrow", mytable; ntasks=1) through Tables.partitions. Reproduce the issue with a lazy iterator whose table exceeds RAM, then verify that the completed behavior processes only one partition in memory at a time.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.