py-why / py-why/EconML

How to use Bootstrap when refitting final stage?

Open
#339 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Hi all,

So basically, roughly I have:

cate = DMLIV(model_Y_X(), model_T_X(), model_T_XZ(),
             dmliv_model_effect(), dmliv_featurizer(),
             n_splits= 20)

##### defining some variable in here
cate.fit(Y, T[:,0], XW, Z, store_final=True)

##### choosing the preferred subset of XW to refit the final in here
cate.refit_final(ph_dmliv_model_effect(), dmliv_featurizer())

So first of all. Should I use BootstrapEstimator in both or just when refitting the final stage as I am guessing (since the final estimates are on the refitted final stage). Secondly, when doing:

BootstrapEstimator(cate, n_bootstrap_samples=100).refit_final(ph_dmliv_model_effect(), dmliv_featurizer())

the following error appears: "unsupported operand type(s) for +: 'DMLIV' and 'DMLIV'" after having finished the replicates.

What's going wrong here? Seems like I have to call an attribute of the class DMLIV like fit or refit, but I am not really sure...
Moreover, is the refitting to be done with the training set? That's because I am specifying fold0, fold1, fold00, folld11 of issue #94 in final model, but I don't know how Bootstrap acts. Moreover, by doing:

boot_est=BootstrapEstimator(est,n_bootstrap_samples=100)
te_pred_interval = boot_est.effect_interval(X_test, lower=1, upper=99)
te_pred_interval

the intervals turn out to be equal to the estimates and equal among each other.

Thank you again!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the DMLIV and BootstrapEstimator usage shown in the issue, including fit, refit_final, effect_interval, and the reference to issue #94. Clarify whether bootstrapping applies during fitting or final-stage refitting, how the reported error arises, how folds and training data are handled, and why the example intervals match the estimates; document the expected usage and results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.