py-why / py-why/EconML

Could you tell me the details of the validity of confidence interval of MetaLearners?

Open
#647 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Background

Thank you for developing such a great library.

I'm reading the document of MetaLearners of EconML.

The document states that the confidence interval is not typically valid:

due to their unrestricted flexibility, they typically do not offer valid confidence intervals, since it is not clear how arbitrary ML methods trade off bias and variance

However, this paper introduced algorithms to estimate confidence interval by bootstrap, and EconML have effect_interval method to estimate confidence interval of MetaLearners.

I also confirmed that effect_interval returns lower and upper bounds of each point.

T_learner.fit(Y, T, X=X, inference="bootstrap")
T_learner.effect_interval(X_test)

Questions

  1. Should we understand that the result of effect_interval could be invalid? How can we interpret the invalidity?
  2. Could you introduce several papers or documents that explain the reason why the confidence interval of MetaLearners could be invalid? (In my understanding, confidence interval estimated by bootstrap could be said to be valid)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked MetaLearners documentation, the cited bootstrap paper, and the effect_interval example in the issue. Determine how the documented caveat relates to bootstrap inference, then clarify when the interval may be unreliable and how users should interpret it; done means the documentation answers both questions with appropriate references.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.