Memory issues when N is increased too much
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Dear
EconML support team.
I am using Causal Forests for calculating CATEs, and the main issue I am facing is that confidence intervals calculated (CIs) are too large with my dataset. After reading QA information, the first solution is increasing the number of trees (nestimators) until the CATEs variance is controlled. The problem is that having ~100K obs, 2K estimators pose large problems of memory. Apart from trying with other hyperparameters, could you provide tips for dealing with it? (I have set up njobs equal to -1).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Causal Forests configuration described here: about 100K observations, 2K estimators, and n_jobs=-1. Reproduce the memory pressure while varying n_estimators and observing confidence-interval variance; done means documenting a reproducible limit and an actionable configuration or confirmed limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100