Arrow.write using a lot of memory to save record batch
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
I am using this function to save a batch Arrow files into a single Arrow file with record batches:
```
function combinebatches(path::String, file::String, batches)
files = Array{String,1}()
for i = 1:batches
push!(files,joinpath(path, "$(file)$(i).arrow"))
end
arrow_parts = Tables.partitioner(Arrow.Table, files)
open(joinpath(path, "$(file)_batched.arrow"), "w") do io
Arrow.write(io, arrow_parts, compress=:zstd)
end
#delete chunks
for i = 1:batches
rm(joinpath(path, "$(file)$(i).arrow"))
end
return nothing
end #combinebatches
```
When saving about 25 uncompressed arrow files with an average size of 1.8 GB each, required a RAM size of around 50GB. As I am partitioning my data to save RAM, this is creating a problem with the high RAM usage (in this case > 50GB). I was expecting that one would not require more than the maximum size of each single file (or perhaps number of threads x file size) to save the record batch.
JULIA_NUM_THREADS is set to 4.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the report with Arrow.write, Tables.partitioner, 25 uncompressed Arrow files, and JULIA_NUM_THREADS=4. Trace how the partitioned input is consumed during writing and measure memory usage; done means combining the files without requiring memory comparable to their total size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100