Causal Forest DML has very wide confidence interval
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Below is my code to estimate treatment effects. There is a much wider confidence interval of ATT (i.e., [-200k, 900k]) by Causal Forest DML model, compared to that calculated by linear DML model (i.e, [200k, 400k]). Are there any ways to make CI by Causal Forest DML narrower, and ideally statistically significant ?
# Linear DML for the ATE
dml = LinearDML(
model_y = rf_reg,
model_t = xgb_class,
discrete_treatment = True,
random_state = 0,
cv = StratifiedKFold(5))
print('Fitting linear DML...')
results_dml = dml.fit(Y=Y,T=T,W=X)
ate = dml.intercept_
ate_lb = dml.intercept__interval()[0]
ate_ub = dml.intercept__interval()[1]
print('DML ATE:', round(ate, 2), 'CI [', round(ate_lb, 2), ',', round(ate_ub, 2), ']', '$')
# Causal Forest for the ITE
cf = CausalForestDML(
model_y = rf_reg,
model_t = xgb_class,
discrete_treatment = True,
cv=StratifiedKFold(5),
random_state = 0,
n_estimators=300,
)
print('Fitting causal forest ...')
results = cf.fit(Y=Y,T=T,X=X,cache_values=True)
# ITE Estimates with lower and upper bound
ite_estimates = cf.effect(X)
lb_estimates, ub_estimates = cf.effect_interval(X)
# Create DataFrame with individual ITE estimates, lower bound, and upper bound
all_individual_effects_df = pd.DataFrame({
'ITE': ite_estimates,
'ITE_lb': lb_estimates,
'ITE_ub': ub_estimates
}, index=df.index)
# Concatenate with other relevant data
all_ITEs = pd.concat([df[['sfdc_customer_id']], T, Y, all_individual_effects_df], axis=1)
# Calculate ATT for treatment group
att = all_ITEs[all_ITEs[treatment_variable] == 1]['ITE'].mean()
att_lb = all_ITEs[all_ITEs[treatment_variable] == 1]['ITE_lb'].mean()
att_ub = all_ITEs[all_ITEs[treatment_variable] == 1]['ITE_ub'].mean()
# Print ATT results
print('CF ATT:', round(att, 2), 'CI [', round(att_lb, 2), ',', round(att_ub, 2), ']', '$')
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file or test is named. Start by reviewing CausalForestDML.effect_interval and the contrast with LinearDML.intercept__interval(), then reproduce the supplied fit configuration; done would require a documented cause or a maintainer-defined change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100