scikit-learn / scikit-learn/scikit-learn
ENH HistGradientBoosting estimators should have `.feature_importances_` attribute
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 67.3k
- Forks
- 27.4k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 58
Description
Describe the workflow you want to enable
The practicality of using the .feature_importances_ attribute to superficially analyze global explanations of your classifier is really useful, IMHO.
Right after evaluating the model, running something like
pd.DataFrame(model.feature_importances_, index=X_train.columns, columns=["feat_imp"]).sort_values("feat_imp", ascending=False)
is the first thing I do.
Furthermore, some variable selection strategies assume that your estimator has this attribute, and it seems suboptimal to me to deprive ourselves of using HistGradientBoosting with these techniques.
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import HistGradientBoostingClassifier
from lightgbm import LGBMClassifier
X = [[ 0.87, -1.34, 0.31 ],
[-2.79, -0.02, -0.85 ],
[-1.34, -0.48, -2.55 ],
[ 1.92, 1.48, 0.65 ]]
y = [0, 1, 0, 1]
selector = SelectFromModel(estimator=LGBMClassifier()).fit(X, y)
selector.get_support()
>>> array([ True, True, True])
selector = SelectFromModel(estimator=HistGradientBoostingClassifier()).fit(X, y)
selector.get_support()
>>> ValueError: when `importance_getter=='auto'`, the underlying estimator HistGradientBoostingClassifier should have `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to feature selector or call fit before calling transform.
I understand the concerns raised by @ogrisel's comment in the original PR that implemented HistGradientBoosting, but I believe we have more points in favor of its implementation than its omission. I'd love to hear your thoughts. :)
Describe your proposed solution
No response
Describe alternatives you've considered, if relevant
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the original HistGradientBoosting implementation PR and the linked discussion about feature importances, then examine HistGradientBoostingClassifier and SelectFromModel behavior. Define and validate the intended feature_importances_ semantics for the estimators, including the example workflow; the issue is done when the agreed behavior is implemented and verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100