JuliaAI / JuliaAI/DecisionTree.jl
better `Leaf` and `Ensemble` structs
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 364
- Forks
- 100
- PR merge metrics
- No merged PRs in 30d
Description
I've implemented GradientBoost and a more efficient version of AdaBoost that interface with treeregressor and treeclassifier and am now looking to port them to the light-weight front facing structs we use, i.e., LeafOrNode and Ensemble.
Trying to do so, I realized that one thing I feel somewhat against is the storage of every labels in the Leaf struct. So I want to discuss if it would be possible to change it so something more lightweight. These values are usually unnecessary, take up too much space, and can be recomputed if needed. For a lot of purposes, a single confidence parameter should suffice (e.g., the impurity of that leaf) And if that is not enough we might add, for example, another function that takes in a set of samples and returns an array containing the indices of the leaves they end up in. This would generalize the current implementation to any dataset and not just the training set, and from that we can very easily rewrite apply_tree_proba.
This may feel like an inconvenience, but for boosting and random forest ensembles that usually use shallow trees, most of the memory cost is in this storage of the samples, and the real inconvenience is dealing with models that are 4GB large or not being able to train your model because you ran out of memory.
One other thing is that the current Ensemble struct doesn't add very much to the current implementation because it's just a list of trees. This means that we have to deal with something like AdaBoost returning both an Ensemble and a list of coefficients that should have been a part of that ensemble in the first place. So I'm also proposing that we encode these coefficients into the model.
tldr; I'm proposing that we use the following structs instead
struct Leaf{S, T}
label :: T
impurity :: Float64
end
struct Ensemble{S, T}
trees :: Vector{Node{S, T}}
coeffs :: Union{Nothing, Vector{Float64}}
method :: String
end
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the existing LeafOrNode and Ensemble definitions and the treeregressor/treeclassifier interfaces first. Compare how GradientBoost, AdaBoost, and apply_tree_proba use stored labels and coefficients; done means agreeing on and implementing the proposed struct API across those callers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100