cannot allocate memory with CausalAnalysis().fit()
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
OS Architecture:
Operating System: Ubuntu 20.04.4 LTS
Kernel: Linux 5.15.0-53-generic
Architecture: x86-64
With a DataFrame of size `(rows, columns) = (9000, 102)`, the `fit` method I often get the cannot allocate memory
from econml.solutions.causal_analysis import CausalAnalysis
import warnings
warnings.simplefilter(action='ignore')
ca = CausalAnalysis(feature_inds=top_features, categorical=categorical,
heterogeneity_inds=None,
classification=True,
nuisance_models="automl",
heterogeneity_model="forest",
n_jobs=-1,
random_state=1234)
ca.fit(X, y)
As a temporary solutuion and some research I have managed a bandaid solution with following linux commands but did not solve the issue
swapon --show
sudo swapoff /swapfile
sudo fallocate -l 5G /swapfile
ls -lh /swapfile
Initially I had `1G `allocated. With the above syntax, now the swap file has `5GB` which did not solve the issue
My issue is if the fit function fails with small dataset with 9000 rows while plenty of the unallocated memory available (I have ram with 64 GB, unused memory about 20GB), has there been similar issue reported?
Some references:
https://stackoverflow.com/questions/5306075/python-memory-allocation-error-using-subprocess-popen
https://stackoverflow.com/questions/20111242/how-to-avoid-errno-12-cannot-allocate-memory-errors-caused-by-using-subprocess
https://stackoverflow.com/questions/20111242/how-to-avoid-errno-12-cannot-allocate-memory-errors-caused-by-using-subprocess
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The reported entry point is CausalAnalysis().fit() with nuisance_models="automl", heterogeneity_model="forest", and n_jobs=-1 on a (9000, 102) DataFrame under Ubuntu 20.04. Reproduce the memory-allocation failure with the supplied configuration; done means identifying a confirmed cause and documenting or testing a corrective change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python
- Domain
- machine-learning, operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100