py-why / py-why/EconML

Performance of First Stage Models

Open
#267 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

How does the predictive performance of the first stage models affect confidence in the final stage model?

For context, I am using the ForestDMLCateEstimator with a random forest regressor and random forest classifier for model y and model t, respectively, and I use a grid search to select hyperparameters for my first stage models. The predictive performance of my model y (regressing the outcome on the features aside from treatment) is low (r-squared is in the vicinity of 0.3) but the performance of model t is fairly higher (classification accuracy is ~0.85). Does the poor performance of my model y negatively affect the validity of my treatment effect results? Is there a typical benchmark for performance of the first stage models?

Thank you in advance and thanks for the great documentation!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ForestDMLCateEstimator documentation and its guidance on first-stage models; review how model y and model t are evaluated in the reported setup. Done means documenting whether the stated R-squared and classification accuracy affect treatment-effect validity and whether a typical performance benchmark exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning, python
Domain
machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.