apache / apache/arrow-julia

Arrow.write using a lot of memory to save record batch

Open
#237 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

I am using this function to save a batch Arrow files into a single Arrow file with record batches:

```
function combinebatches(path::String, file::String, batches)
files = Array{String,1}()
for i = 1:batches
push!(files,joinpath(path, "$(file)$(i).arrow"))
end
arrow_parts = Tables.partitioner(Arrow.Table, files)
open(joinpath(path, "$(file)_batched.arrow"), "w") do io
Arrow.write(io, arrow_parts, compress=:zstd)
end
#delete chunks
for i = 1:batches
rm(joinpath(path, "$(file)$(i).arrow"))
end
return nothing
end #combinebatches
```
When saving about 25 uncompressed arrow files with an average size of 1.8 GB each, required a RAM size of around 50GB. As I am partitioning my data to save RAM, this is creating a problem with the high RAM usage (in this case > 50GB). I was expecting that one would not require more than the maximum size of each single file (or perhaps number of threads x file size) to save the record batch.

JULIA_NUM_THREADS is set to 4.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the report with Arrow.write, Tables.partitioner, 25 uncompressed Arrow files, and JULIA_NUM_THREADS=4. Trace how the partitioned input is consumed during writing and measure memory usage; done means combining the files without requiring memory comparable to their total size.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.