py-why / py-why/EconML

Weird result with CausalForestDML

Open
#584 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Hi, I am experimenting with the great EconML package but I am having some weird results with the function CausalForestDML.
I am using the same code from this notebook with data from Duflo, Dupas and Kremer (2011) Peer Effects, Teacher Incentives, and the Impact of Tracking, a clustered RCT in Kenya.

I wanted to compare the ATE results for CausalForestDML and ForestDRLearner, while the latter method provides a plausible estimate of the ATE (0.13 against the 0.11 Neyman estimator), the former gives very weird estimates of the ATE (from 500 standard deviations to 0.35). I was able to figure that the issue seems to be in the interaction between CausalForestDML and the model for the propensity score chosen by GridSearchCVList (in my case the GradientBoostingClassifier). If I use as the propensity score model the GradientBoostingClassifier I recover the weird ATE of 0.35, instead if I use a RandomForestClassifier I get a better estimate of 0.15, lastly if I use the standard LogisticRegression I get an ATE of 0.12.

I guess my question is why does this happen? Why does the "best" model, according to GridSearchCVList, performs so poorly, while the "worst" performs the best when used with CausalForestDML? And lastly, why does this weird phenomenon not happen with the ForestDRLearner?

I attach the data and the code I am using, thanks a lot for any feedback. This package is awesome and it will be of great help for many.

Merged_nopercentile_nomissing.csv

import pandas as pd
import numpy as np
np.set_printoptions(threshold=np.inf)
np.random.seed(11397)

#%% Import data
df = pd.read_csv('./Merged_nopercentile_nomissing.csv')

X = df.loc[:,'girl':'district2']
X.drop('percentile', inplace=True, axis=1)
T = df.loc[:, 'tracking']
Y = df.loc[:, 'totalscore_std']

from econml.sklearn_extensions.model_selection import GridSearchCVList
from sklearn.linear_model import LassoCV, LogisticRegressionCV, Lasso, LogisticRegression
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.base import clone

def first_stage_reg():
    return GridSearchCVList([Lasso(),
                             RandomForestRegressor(n_estimators=1000, random_state=123),
                             GradientBoostingRegressor(n_estimators=1000 , random_state=123)],
                             param_grid_list=[{'alpha': [.001, .01, .1, 1, 10]},
                                               {'max_depth': [3, None],
                                               'min_samples_leaf': [10, 50]},
                                              {'n_estimators': [50, 100],
                                               'max_depth': [3],
                                               'min_samples_leaf': [10, 30]}],
                             cv=5,
                             scoring='neg_mean_squared_error')

def first_stage_clf():
    return GridSearchCVList([LogisticRegression(),
                             RandomForestClassifier(random_state=123),
                             GradientBoostingClassifier(random_state=123)],
                             param_grid_list=[{'C': [0.01, .1, 1, 10, 100]},
                                              {'max_depth': [3, 5],
                                               'min_samples_leaf': [10, 50]},
                                              {'max_depth': [3],
                                               'min_samples_leaf': [10, 30]}],
                             cv=5,
                             scoring='neg_mean_squared_error')

model_y = clone(first_stage_reg().fit(X, Y).best_estimator_)
model_t = clone(first_stage_clf().fit(X, T).best_estimator_)

from econml.dml import CausalForestDML
est = CausalForestDML(model_t = model_t, #LogisticRegression(),
                      model_y = model_y, #LassoCV(),
                      discrete_treatment=True,
                      cv=5,
                      n_estimators= 2000,
                      random_state=123)
est.tune(Y, T, X=X).fit(Y, T, X=X, cache_values=True)
#est.summary()
ate = est.ate(X)


from econml.dr import ForestDRLearner
est2 = ForestDRLearner(model_propensity = model_t, #GradientBoostingClassifier(),
                      model_regression = model_y, #LassoCV(),
                      cv=5,
                      n_estimators= 2000,
                      random_state=123).fit(Y, T, X=X, cache_values=True)
ate2 = est2.ate(X)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported estimates using the attached CSV and the code from ForestLearners Basic Example.ipynb. Start with GridSearchCVList, CausalForestDML, and ForestDRLearner, comparing the three propensity models and their ATE results. Done means explaining the discrepancy or identifying a reproducible implementation issue and documenting the relevant behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.