scikit-learn / scikit-learn/scikit-learn

Add random forest row-subsampling without replacement

Open
#20,953 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module:ensemble Needs Decision New Feature
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

Describe the workflow you want to enable

I do appreciate the current option of disabling bootstrapping via the Boolean argument bootstrap.
However, currently there is only one alternative: If False, the whole (identical) dataset is used to build each tree.
There is well known research though (starting with the paper by Strobl et al. from 2007) that in certain situations subsamples drawn without replacement leads to better performance.
Many well known random forest implementations (such as ranger ) offer this sub sampling as an alternative to the bootstrap.

I would greatly appreciate it if sklearn would offer the same.

Describe your proposed solution

For both RandomForestRegressor and RandomForestClassifier allow the user to draw subsamples without replacement for each tree instead of the bootstrap.
The user can choose the fraction of sub samples drawn for each tree (default: 0.632 )
Ideally the functions _generate_unsampled_indices and _generate_sampled_indices would still work.

Describe alternatives you've considered, if relevant

No response

Additional context

No response

API considerations

Currently, random forests have the option bootstap=True or False. If this new feature of tree-wise row-subsampling without replacement is added, there are several options:

  1. Add a new option with_replacement=True (default), or False, that only takes effect if bootstrap=True.
    Disadvantage: The term bootstrap explicitly means sampling with replacement.
  2. Add a new option row_subsampling=True (default) or False, which samples with replacement if bootstrap=True and without replacement if bootstrap=False.
    Disadvantage: It would change current behaviour for bootstrap=False, which currently means no sampling at all.
  3. Add new option sampling="bootstrap" (default), and allow callable / splitter to be passed. Deprecate option bootstrap (proposed in https://github.com/scikit-learn/scikit-learn/issues/20953#issuecomment-923957749).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review RandomForestRegressor and RandomForestClassifier, along with the _generate_unsampled_indices and _generate_sampled_indices helpers named in the issue. Compare the proposed sampling API alternatives and establish tests for per-tree sampling without replacement, configurable sample fractions, and preserved existing bootstrap behavior; done means both estimators support the agreed API and behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.