JuliaAI / JuliaAI/DecisionTree.jl

Memory leakage upon repeated training

Open
#230 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
364
Forks
100
PR merge metrics
No merged PRs in 30d

Description

Hi, I've been creating some random forest regressors lately and I've noticed high memory usage during hyperparameter tuning. It turns out that there is some memory leakage in the package. For some reason Julia does not delete the trees when they become unreachable.

Following is a MWE: after finishing run_forests, some memory should be reclaimed but it doesn't happen and memory usage increases. When running the second loop however, memory usage stays constant.

using DecisionTree
function run_forests(features, labels)
    forest = build_forest(labels, features)
    labels .+= apply_forest(forest, features)
    labels ./= 2
end

function run_something_else(features, labels)
    C = repeat(features, inner=(2,2))
    labels ./= vec(sum(C, dims=2))[1:length(labels)]
end

const features = rand(10_000, 10)
const labels = sum(features, dims=2) |> vec

# notice memory consumption increases every couple of iterations
for i = 1:1_000
    run_forests(features, labels)
    @info "Iteration $i current memory used" Sys.maxrss()
end

# notice memory consumption does not increase every couple of iterations
for i = 1:1_000
    run_something_else(features, labels)
    @info "Iteration $i current memory used" Sys.maxrss()
end

Any idea what might cause this?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the Julia MWE with DecisionTree, focusing on repeated calls to build_forest and apply_forest in run_forests. Compare memory behavior with run_something_else and investigate why trees are not reclaimed after run_forests returns. Done means identifying and correcting the reported memory growth while preserving the stable behavior of the second loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.