JuliaAI / JuliaAI/DecisionTree.jl

Excessive memory usage

Open
#123 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
364
Forks
100
PR merge metrics
No merged PRs in 30d

Description

I have a data set of dimensions (87390, 243). Most of the columns are categorical variables that have been one-hot encoded. The size of the data set in memory is ~160 MB. I compared the memory usage for DecisionTree.jl and R's ranger package.

DecisionTree.jl

using DecisionTree

df = CSV.read("rf_training_data.csv")

y = string.(df.y)
X = convert(Matrix, df[:, 2:end])

n_subfeatures = 15
n_trees = 600

# Default vaues:
# partial_sampling = 0.7
# max_depth = -1
# min_samples_leaf = 1

rf = build_forest(y, X, n_subfeatures, n_trees)

Memory consumption:

julia> varinfo(r"rf")
  name      size summary                 
  –––– ––––––––– ––––––––––––––––––––––––
  rf   1.417 GiB Ensemble{Float64,String}

ranger

library(readr)
library(ranger)

df <- read_csv('rf_training_data.csv')
df$y <- factor(df$y)

rf <- ranger(
    y ~ .,
    data = df,
    num.trees = 600,
    mtry = 15,
    min.node.size = 1,
    replace = FALSE,
    sample.fraction = 0.7
)

Memory consumption:

> print(object.size(rf), units = "MB")
585.2 Mb

Conclusion

Thus, it appears that DecisionTree.jl is using 2.4x as much memory as ranger for this model. Is it possible to reduce the memory footprint of DecisionTree.jl? I can provide a scrubbed version of my data set if that helps.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Julia and R memory comparison from the issue with the stated data dimensions and random-forest settings. Trace the model-building path responsible for the larger allocation, then document or implement a reduction in memory usage and rerun the comparison to verify the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia, r
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.