scikit-learn / scikit-learn/scikit-learn

ENH HistGradientBoosting estimators should have `.feature_importances_` attribute

Open
#25,707 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

API Needs Decision New Feature
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

Describe the workflow you want to enable

The practicality of using the .feature_importances_ attribute to superficially analyze global explanations of your classifier is really useful, IMHO.

Right after evaluating the model, running something like

pd.DataFrame(model.feature_importances_, index=X_train.columns, columns=["feat_imp"]).sort_values("feat_imp", ascending=False)

is the first thing I do.

Furthermore, some variable selection strategies assume that your estimator has this attribute, and it seems suboptimal to me to deprive ourselves of using HistGradientBoosting with these techniques.

from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import HistGradientBoostingClassifier
from lightgbm import LGBMClassifier

X = [[ 0.87, -1.34,  0.31 ],
     [-2.79, -0.02, -0.85 ],
     [-1.34, -0.48, -2.55 ],
     [ 1.92,  1.48,  0.65 ]]
y = [0, 1, 0, 1]
selector = SelectFromModel(estimator=LGBMClassifier()).fit(X, y)
selector.get_support()
>>> array([ True,  True,  True])

selector = SelectFromModel(estimator=HistGradientBoostingClassifier()).fit(X, y)
selector.get_support()
>>> ValueError: when `importance_getter=='auto'`, the underlying estimator HistGradientBoostingClassifier should have `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to feature selector or call fit before calling transform.

I understand the concerns raised by @ogrisel's comment in the original PR that implemented HistGradientBoosting, but I believe we have more points in favor of its implementation than its omission. I'd love to hear your thoughts. :)

Describe your proposed solution

No response

Describe alternatives you've considered, if relevant

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the original HistGradientBoosting implementation PR and the linked discussion about feature importances, then examine HistGradientBoostingClassifier and SelectFromModel behavior. Define and validate the intended feature_importances_ semantics for the estimators, including the example workflow; the issue is done when the agreed behavior is implemented and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.