py-why / py-why/EconML

DRtester does not work for binary treatment AND binary outcome

Open
#875 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Hi, here is the code to reproduce the error:

import numpy as np
import pandas as pd
import scipy.stats as st

from econml.metalearners import TLearner
from econml.dml import DML

from econml.validate.drtester import DRtester

from sklearn.ensemble import (RandomForestClassifier, RandomForestRegressor,
                              GradientBoostingClassifier, GradientBoostingRegressor, 
                              AdaBoostClassifier, AdaBoostRegressor)

from sklearn.svm import SVC, SVR
from sklearn.neural_network import MLPClassifier, MLPRegressor
from lightgbm import LGBMClassifier, LGBMRegressor
                                
    # Classifiers
from sklearn.dummy import DummyClassifier

np.random.seed(123)

N = 20000  # number of units
K = 5  # number of covariates
num_treatments = 2 # number of treatments (excluding control)

# Generate random Xs
X_mu = np.zeros(5)  # Means of Xs
# Random covariance matrix of Xs
X_sig = np.diag(np.random.rand(5))
X = st.multivariate_normal(X_mu, X_sig).rvs(N)

# Effect of Xs on outcome
X_beta = np.random.uniform(0, 5, K)
# Effect of treatment on outcomes
D_beta = np.array([0, 1, 2])
# Effect of treatment on outcome conditional on X1
DX1_beta = np.array([0, 0, 3])

# Generate treatments based on X and random noise
beta_treat = np.random.uniform(-1, 1, (num_treatments + 1, K))
D1 = np.zeros((N, num_treatments + 1))
for k in range(num_treatments + 1):
    D1[:, k] = X @ beta_treat[k, :] + np.random.gumbel(0, 1, N)
D = np.array([np.where(D1[i, :] == np.max(D1[i, :]))[0][0] for i in range(N)])
D_dum = pd.get_dummies(D)

# Generate Y (based on X, D, and random noise)
Y_sig = 1  # Variance of random outcome noise
Y = X @ X_beta + (D_dum @ D_beta) + X[:, 1] * (D_dum @ DX1_beta) + np.random.normal(0, Y_sig, N)
Y = Y.to_numpy()

# Split into training/validation samples
train_prop = .5
train_N = np.ceil(train_prop * N)
ind = np.array(range(N))
train_ind = np.random.choice(N, int(train_N), replace=False)
val_ind = ind[~np.isin(ind, train_ind)]

Xtrain, Dtrain, Ytrain = X[train_ind], D[train_ind], Y[train_ind]
Xval, Dval, Yval = X[val_ind], D[val_ind], Y[val_ind]



model_regression = GradientBoostingRegressor(random_state=0)
model_propensity = RandomForestClassifier(random_state=0)

nval = len(Dval)
ntrain = len(Dtrain)
Dval = np.random.choice(2, nval)
Dtrain = np.random.choice(2, ntrain)

nval = len(Yval)
ntrain = len(Ytrain)
Yval = np.random.choice(2, nval)
Ytrain = np.random.choice(2, ntrain)

est_dm = DML(model_y = model_propensity, 
             model_t= model_propensity,
             model_final = model_regression,
             discrete_outcome=True,
             discrete_treatment=True,
             cv=5)

est_dm.fit(Ytrain, Dtrain, X=Xtrain)

# Initialize DRTester and fit/predict nuisance models
dml_tester = DRtester(
    model_regression=model_regression, 
    model_propensity=model_propensity,
    cate=est_dm
).fit_nuisance(Xval, Dval, Yval, Xtrain, Dtrain, Ytrain)

res_dml = dml_tester.evaluate_all(Xval, Xtrain)
res_dml.summary()

Error message:

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_28104/1480639724.py in <module>
     92 ).fit_nuisance(Xval, Dval, Yval, Xtrain, Dtrain, Ytrain)
     93 
---> 94 res_dml = dml_tester.evaluate_all(Xval, Xtrain)
     95 res_dml.summary()

[~/.local/lib/python3.10/site-packages/econml/validate/drtester.py](http://localhost:8888/home/yvdc/.local/lib/python3.10/site-packages/econml/validate/drtester.py) in evaluate_all(self, Xval, Xtrain, n_groups)
    594             self.get_cate_preds(Xval, Xtrain)
    595 
--> 596         blp_res = self.evaluate_blp()
    597         cal_res = self.evaluate_cal(n_groups=n_groups)
    598         qini_res = self.evaluate_uplift(metric='qini')

[~/.local/lib/python3.10/site-packages/econml/validate/drtester.py](http://localhost:8888/home/yvdc/.local/lib/python3.10/site-packages/econml/validate/drtester.py) in evaluate_blp(self, Xval, Xtrain)
    462 
    463         if self.n_treat == 1:  # binary treatment
--> 464             reg = OLS(self.dr_val_, add_constant(self.cate_preds_val_)).fit()
    465             params = [reg.params[1]]
    466             errs = [reg.bse[1]]

[~/.local/lib/python3.10/site-packages/statsmodels/tools/tools.py](http://localhost:8888/home/yvdc/.local/lib/python3.10/site-packages/statsmodels/tools/tools.py) in add_constant(data, prepend, has_constant)
    191         x = x[:, None]
    192     elif x.ndim > 2:
--> 193         raise ValueError('Only implemented for 2-dimensional arrays')
    194 
    195     is_nonzero_const = np.ptp(x, axis=0) == 0

ValueError: Only implemented for 2-dimensional arrays

It does work when the outcome y is continuous.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the provided binary-treatment and binary-outcome example, then inspect econml/validate/drtester.py at evaluate_all and evaluate_blp, along with the statsmodels add_constant call in the traceback. Determine why the CATE predictions produce an unsupported shape, and verify that evaluate_all completes for binary outcomes while the existing continuous-outcome behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python, scikit-learn
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.