JuliaAI / JuliaAI/DecisionTree.jl
Memory leakage upon repeated training
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 364
- Forks
- 100
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I've been creating some random forest regressors lately and I've noticed high memory usage during hyperparameter tuning. It turns out that there is some memory leakage in the package. For some reason Julia does not delete the trees when they become unreachable.
Following is a MWE: after finishing run_forests, some memory should be reclaimed but it doesn't happen and memory usage increases. When running the second loop however, memory usage stays constant.
using DecisionTree
function run_forests(features, labels)
forest = build_forest(labels, features)
labels .+= apply_forest(forest, features)
labels ./= 2
end
function run_something_else(features, labels)
C = repeat(features, inner=(2,2))
labels ./= vec(sum(C, dims=2))[1:length(labels)]
end
const features = rand(10_000, 10)
const labels = sum(features, dims=2) |> vec
# notice memory consumption increases every couple of iterations
for i = 1:1_000
run_forests(features, labels)
@info "Iteration $i current memory used" Sys.maxrss()
end
# notice memory consumption does not increase every couple of iterations
for i = 1:1_000
run_something_else(features, labels)
@info "Iteration $i current memory used" Sys.maxrss()
end
Any idea what might cause this?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the Julia MWE with DecisionTree, focusing on repeated calls to build_forest and apply_forest in run_forests. Compare memory behavior with run_something_else and investigate why trees are not reclaimed after run_forests returns. Done means identifying and correcting the reported memory growth while preserving the stable behavior of the second loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100