JuliaAI / JuliaAI/DecisionTree.jl
Excessive memory usage
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 364
- Forks
- 100
- PR merge metrics
- No merged PRs in 30d
Description
I have a data set of dimensions (87390, 243). Most of the columns are categorical variables that have been one-hot encoded. The size of the data set in memory is ~160 MB. I compared the memory usage for DecisionTree.jl and R's ranger package.
DecisionTree.jl
using DecisionTree
df = CSV.read("rf_training_data.csv")
y = string.(df.y)
X = convert(Matrix, df[:, 2:end])
n_subfeatures = 15
n_trees = 600
# Default vaues:
# partial_sampling = 0.7
# max_depth = -1
# min_samples_leaf = 1
rf = build_forest(y, X, n_subfeatures, n_trees)
Memory consumption:
julia> varinfo(r"rf")
name size summary
–––– ––––––––– ––––––––––––––––––––––––
rf 1.417 GiB Ensemble{Float64,String}
ranger
library(readr)
library(ranger)
df <- read_csv('rf_training_data.csv')
df$y <- factor(df$y)
rf <- ranger(
y ~ .,
data = df,
num.trees = 600,
mtry = 15,
min.node.size = 1,
replace = FALSE,
sample.fraction = 0.7
)
Memory consumption:
> print(object.size(rf), units = "MB")
585.2 Mb
Conclusion
Thus, it appears that DecisionTree.jl is using 2.4x as much memory as ranger for this model. Is it possible to reduce the memory footprint of DecisionTree.jl? I can provide a scrubbed version of my data set if that helps.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Julia and R memory comparison from the issue with the stated data dimensions and random-forest settings. Trace the model-building path responsible for the larger allocation, then document or implement a reduction in memory usage and rerun the comparison to verify the result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia, r
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100