apache / apache/arrow-julia

Memory leak(?) in `Arrow.write()`

Open
#247 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

@aminnj noticed this first, download the `Run2012...` file from http://opendata.web.cern.ch/record/12341/files/Run2012BC_DoubleMuParked_Muons.root and the setup:
```julia
# call this test_partition.jl
using UnROOT, Tables, Arrow ; const tf = LazyTree(ROOTFile("Run2012BC_DoubleMuParked_Muons.root"),"Events");

function _lockedget(t::LazyTree, r::UnitRange)
f = getproperty(t,first(propertynames(t))).f
lock(f)
@show r
GC.gc()
try
return t[r]
catch
finally
unlock(f)
end
end

Tables.partitions(t::LazyTree) = (_lockedget(t, r) for r in UnROOT._clusterranges(t))
```

Then, setting memory limit to 3GB:
```
$ ulimit -Sv 3000000
```

then run one of these:
```julia
include("test_partition.jl")

# this finishes
for t in Tables.partitions(tf)
@show length(t)
end
```

```julia
include("test_partition.jl")

# this crashes
tf |> Arrow.write("doublemu.arrow", ntasks=1)
```

So `UnROOT.jl` on it's own isn't leaking memory, so maybe we're doing something that doesn't play naively with Arrow?

Contributor guide

No contributing guide indexed for this repository

Research direction

Download Run2012BC_DoubleMuParked_Muons.root and run the provided test_partition.jl setup under the 3GB ulimit. Compare iterating Tables.partitions(tf) with tf |> Arrow.write("doublemu.arrow", ntasks=1), then determine whether the crash is caused by the Arrow.write interaction and document a reproducible outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.