Performance of First Stage Models
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
How does the predictive performance of the first stage models affect confidence in the final stage model?
For context, I am using the ForestDMLCateEstimator with a random forest regressor and random forest classifier for model y and model t, respectively, and I use a grid search to select hyperparameters for my first stage models. The predictive performance of my model y (regressing the outcome on the features aside from treatment) is low (r-squared is in the vicinity of 0.3) but the performance of model t is fairly higher (classification accuracy is ~0.85). Does the poor performance of my model y negatively affect the validity of my treatment effect results? Is there a typical benchmark for performance of the first stage models?
Thank you in advance and thanks for the great documentation!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ForestDMLCateEstimator documentation and its guidance on first-stage models; review how model y and model t are evaluated in the reported setup. Done means documenting whether the stated R-squared and classification accuracy affect treatment-effect validity and whether a typical performance benchmark exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning, python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100