Memory leak(?) in `Arrow.write()`
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
@aminnj noticed this first, download the `Run2012...` file from http://opendata.web.cern.ch/record/12341/files/Run2012BC_DoubleMuParked_Muons.root and the setup:
```julia
# call this test_partition.jl
using UnROOT, Tables, Arrow ; const tf = LazyTree(ROOTFile("Run2012BC_DoubleMuParked_Muons.root"),"Events");
function _lockedget(t::LazyTree, r::UnitRange)
f = getproperty(t,first(propertynames(t))).f
lock(f)
@show r
GC.gc()
try
return t[r]
catch
finally
unlock(f)
end
end
Tables.partitions(t::LazyTree) = (_lockedget(t, r) for r in UnROOT._clusterranges(t))
```
Then, setting memory limit to 3GB:
```
$ ulimit -Sv 3000000
```
then run one of these:
```julia
include("test_partition.jl")
# this finishes
for t in Tables.partitions(tf)
@show length(t)
end
```
```julia
include("test_partition.jl")
# this crashes
tf |> Arrow.write("doublemu.arrow", ntasks=1)
```
So `UnROOT.jl` on it's own isn't leaking memory, so maybe we're doing something that doesn't play naively with Arrow?
Contributor guide
No contributing guide indexed for this repository
Research direction
Download Run2012BC_DoubleMuParked_Muons.root and run the provided test_partition.jl setup under the 3GB ulimit. Compare iterating Tables.partitions(tf) with tf |> Arrow.write("doublemu.arrow", ntasks=1), then determine whether the crash is caused by the Arrow.write interaction and document a reproducible outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100