Weird result with CausalForestDML
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I am experimenting with the great EconML package but I am having some weird results with the function CausalForestDML.
I am using the same code from this notebook with data from Duflo, Dupas and Kremer (2011) Peer Effects, Teacher Incentives, and the Impact of Tracking, a clustered RCT in Kenya.
I wanted to compare the ATE results for CausalForestDML and ForestDRLearner, while the latter method provides a plausible estimate of the ATE (0.13 against the 0.11 Neyman estimator), the former gives very weird estimates of the ATE (from 500 standard deviations to 0.35). I was able to figure that the issue seems to be in the interaction between CausalForestDML and the model for the propensity score chosen by GridSearchCVList (in my case the GradientBoostingClassifier). If I use as the propensity score model the GradientBoostingClassifier I recover the weird ATE of 0.35, instead if I use a RandomForestClassifier I get a better estimate of 0.15, lastly if I use the standard LogisticRegression I get an ATE of 0.12.
I guess my question is why does this happen? Why does the "best" model, according to GridSearchCVList, performs so poorly, while the "worst" performs the best when used with CausalForestDML? And lastly, why does this weird phenomenon not happen with the ForestDRLearner?
I attach the data and the code I am using, thanks a lot for any feedback. This package is awesome and it will be of great help for many.
Merged_nopercentile_nomissing.csv
import pandas as pd
import numpy as np
np.set_printoptions(threshold=np.inf)
np.random.seed(11397)
#%% Import data
df = pd.read_csv('./Merged_nopercentile_nomissing.csv')
X = df.loc[:,'girl':'district2']
X.drop('percentile', inplace=True, axis=1)
T = df.loc[:, 'tracking']
Y = df.loc[:, 'totalscore_std']
from econml.sklearn_extensions.model_selection import GridSearchCVList
from sklearn.linear_model import LassoCV, LogisticRegressionCV, Lasso, LogisticRegression
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.base import clone
def first_stage_reg():
return GridSearchCVList([Lasso(),
RandomForestRegressor(n_estimators=1000, random_state=123),
GradientBoostingRegressor(n_estimators=1000 , random_state=123)],
param_grid_list=[{'alpha': [.001, .01, .1, 1, 10]},
{'max_depth': [3, None],
'min_samples_leaf': [10, 50]},
{'n_estimators': [50, 100],
'max_depth': [3],
'min_samples_leaf': [10, 30]}],
cv=5,
scoring='neg_mean_squared_error')
def first_stage_clf():
return GridSearchCVList([LogisticRegression(),
RandomForestClassifier(random_state=123),
GradientBoostingClassifier(random_state=123)],
param_grid_list=[{'C': [0.01, .1, 1, 10, 100]},
{'max_depth': [3, 5],
'min_samples_leaf': [10, 50]},
{'max_depth': [3],
'min_samples_leaf': [10, 30]}],
cv=5,
scoring='neg_mean_squared_error')
model_y = clone(first_stage_reg().fit(X, Y).best_estimator_)
model_t = clone(first_stage_clf().fit(X, T).best_estimator_)
from econml.dml import CausalForestDML
est = CausalForestDML(model_t = model_t, #LogisticRegression(),
model_y = model_y, #LassoCV(),
discrete_treatment=True,
cv=5,
n_estimators= 2000,
random_state=123)
est.tune(Y, T, X=X).fit(Y, T, X=X, cache_values=True)
#est.summary()
ate = est.ate(X)
from econml.dr import ForestDRLearner
est2 = ForestDRLearner(model_propensity = model_t, #GradientBoostingClassifier(),
model_regression = model_y, #LassoCV(),
cv=5,
n_estimators= 2000,
random_state=123).fit(Y, T, X=X, cache_values=True)
ate2 = est2.ate(X)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported estimates using the attached CSV and the code from ForestLearners Basic Example.ipynb. Start with GridSearchCVList, CausalForestDML, and ForestDRLearner, comparing the three propensity models and their ATE results. Done means explaining the discrepancy or identifying a reproducible implementation issue and documenting the relevant behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100