High memory footprint for big dataframes in CausalForest model
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
I'm facing multiple memory issues when using CausalForestDML class with big dataframes.
In both the fit method and the effect / effect inference methods, larger dataframes (even not so large, ex: 2M rows / 20 columns) tend to have an exponential memory occupation (~180GB ram).
We could start an investigation inside the code to try to minimize the memory footprint / optimize both methods below:
- predict_point_and_var, from https://github.com/py-why/EconML/blob/db1e254f1e5cd476b917cfe1c2984db4c264bb66/econml/grf/_base_grf.py#L650, used in both effect / effect_inference methods
- fit, from https://github.com/py-why/EconML/blob/db1e254f1e5cd476b917cfe1c2984db4c264bb66/econml/grf/_base_grf.py#L192
My first investigations are tending to indicate that problem is originating from these methods above.
Any clues? @kbattocchi
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read econml/grf/_base_grf.py at fit and predict_point_and_var, then reproduce the reported memory growth with a large dataframe through fit, effect, and effect_inference. Trace which operations account for the footprint and validate that the relevant methods use substantially less memory without changing their results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100