py-why / py-why/EconML

[question] Can't replicate manual DML results in EconML

Open
#958 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Hello, I am a novice in EconML Library. I am comparing the DML with single treatment variable results using EconML vs coding it manually and getting wildly different results, with manually coded DML OLS model resulting in much higher R2 score. Here's what I do:

  • Log-transform treatment and effect variables
  • Take zscore of all other input features (they are all numeric)
  • Use DML with LightGBMRegressor for both model_y and model_t (EconML) with leave-one-out cross-validation (input size-1, also tried 2, 5, 10 etc.) Use lgbm.cv with input size-1 (manual). Both are used with default hyperparameters.
  • Use StatsModelsLinearRegression for regression (EconML). Use ols from statsmodels.formula.api for regression (Manual)
  • Also tried using ols from statsmodels.formula.api for regression on the residuals that I get from EconML DML estimate.

I don't expect exactly the same results, but I what I am getting is too different - R2=0.01 (EconML) R2=0.55 (manual).

I can't include the code because of data restrictions, but, if it will help, can try it out on sample data and see if this behaviour persists. Before I do that though, perhaps there is something obviously wrong with my process. Please let me know if so.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with a minimal sample-data reproduction comparing EconML DML using LightGBMRegressor and StatsModelsLinearRegression against the described manual OLS and residual-based approaches. Check the treatment and outcome transformations, standardization, cross-validation settings, and default model parameters; done means identifying whether the R2 discrepancy is expected or reproducible and documenting the cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.