[question] Can't replicate manual DML results in EconML
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I am a novice in EconML Library. I am comparing the DML with single treatment variable results using EconML vs coding it manually and getting wildly different results, with manually coded DML OLS model resulting in much higher R2 score. Here's what I do:
- Log-transform treatment and effect variables
- Take zscore of all other input features (they are all numeric)
- Use DML with LightGBMRegressor for both model_y and model_t (EconML) with leave-one-out cross-validation (input size-1, also tried 2, 5, 10 etc.) Use lgbm.cv with input size-1 (manual). Both are used with default hyperparameters.
- Use StatsModelsLinearRegression for regression (EconML). Use ols from statsmodels.formula.api for regression (Manual)
- Also tried using ols from statsmodels.formula.api for regression on the residuals that I get from EconML DML estimate.
I don't expect exactly the same results, but I what I am getting is too different - R2=0.01 (EconML) R2=0.55 (manual).
I can't include the code because of data restrictions, but, if it will help, can try it out on sample data and see if this behaviour persists. Before I do that though, perhaps there is something obviously wrong with my process. Please let me know if so.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with a minimal sample-data reproduction comparing EconML DML using LightGBMRegressor and StatsModelsLinearRegression against the described manual OLS and residual-based approaches. Check the treatment and outcome transformations, standardization, cross-validation settings, and default model parameters; done means identifying whether the R2 discrepancy is expected or reproducible and documenting the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100