Could you tell me the details of the validity of confidence interval of MetaLearners?
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Background
Thank you for developing such a great library.
I'm reading the document of MetaLearners of EconML.
The document states that the confidence interval is not typically valid:
due to their unrestricted flexibility, they typically do not offer valid confidence intervals, since it is not clear how arbitrary ML methods trade off bias and variance
However, this paper introduced algorithms to estimate confidence interval by bootstrap, and EconML have effect_interval method to estimate confidence interval of MetaLearners.
I also confirmed that effect_interval returns lower and upper bounds of each point.
T_learner.fit(Y, T, X=X, inference="bootstrap")
T_learner.effect_interval(X_test)
Questions
- Should we understand that the result of
effect_intervalcould be invalid? How can we interpret the invalidity? - Could you introduce several papers or documents that explain the reason why the confidence interval of MetaLearners could be invalid? (In my understanding, confidence interval estimated by bootstrap could be said to be valid)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked MetaLearners documentation, the cited bootstrap paper, and the effect_interval example in the issue. Determine how the documented caveat relates to bootstrap inference, then clarify when the interval may be unreliable and how users should interpret it; done means the documentation answers both questions with appropriate references.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100