`Arrow.write` doesn't play nicely when the target table is lazy.
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Imagine a table `mytable` ~ 10x RAM size, the `Tables.partitions` is naturally defined as a lazy iterator:
```julia
Tables.partitions(t::MyTable) = (t[range] for range in range_partition(t))
```
But when subsequently calling: `Arrow.write("out.arrow", mytable; ntasks=1)`, because the:
https://github.com/JuliaData/Arrow.jl/blob/967e3c67437219698afae889a13e881b60033fe3/src/write.jl#L127
the table is materialized in RAM anyway (because `t[range]` triggers materialization). Is there any blessed way to make this a truly sequential process such that we never hold more than one partition in RAM at any time?
related: https://github.com/tamasgal/UnROOT.jl/issues/114#issuecomment-925401278
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at src/write.jl around line 127 and trace Arrow.write("out.arrow", mytable; ntasks=1) through Tables.partitions. Reproduce the issue with a lazy iterator whose table exceeds RAM, then verify that the completed behavior processes only one partition in memory at a time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100