Tips on tuning DROrthoForest - is its extreme slowness during evaluation expected?
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4.8k
- Forks
- 827
- PR merge metrics
- No merged PRs in 30d
Description
Can you please provide some tips on tuning DROrthoForest? From its description it sounds like it would be especially useful on larger datasets due to a local component to the fit, but its performance (in terms of speed) when I tried that is terrible.
I fit it on a dataset with 100K data points and some 600 variables (most of them one-hot-encoded categoricals), and the fit itself is fast enough (about 20 mins), but evaluating the model on that same dataset has so far run 48 hours and shows no signs of completing.
Some tests show that the max_depth and n_trees parameters have almost no effect on speed of evaluation.
First of all, is this intended behavior? It seems weird that it takes orders of magnitude longer to evaluate a forest-style model than to fit it.
Secondly, how can I tweak it so the model becomes usable on datasets of one-two million rows?
Thanks a lot!
fitting backdoor.econml.orf.DROrthoForest {'init_params': {'propensity_model': DummyClassifier(), 'model_Y': Ridge(alpha=0.01), 'n_jobs': 15, 'max_depth': 10, 'n_trees': 500, 'min_leaf_size': 1258, 'backend': 'threading'}, 'fit_params': {}}
[Parallel(n_jobs=15)]: Using backend ThreadingBackend with 15 concurrent workers.
[Parallel(n_jobs=15)]: Done 2 tasks | elapsed: 19.0s
[Parallel(n_jobs=15)]: Done 98 tasks | elapsed: 2.2min
[Parallel(n_jobs=15)]: Done 258 tasks | elapsed: 5.5min
[Parallel(n_jobs=15)]: Done 500 out of 500 | elapsed: 10.4min finished
[Parallel(n_jobs=15)]: Using backend ThreadingBackend with 15 concurrent workers.
[Parallel(n_jobs=15)]: Done 2 tasks | elapsed: 18.6s
[Parallel(n_jobs=15)]: Done 98 tasks | elapsed: 2.1min
[Parallel(n_jobs=15)]: Done 258 tasks | elapsed: 5.5min
[Parallel(n_jobs=15)]: Done 500 out of 500 | elapsed: 10.4min finished
[Parallel(n_jobs=15)]: Using backend ThreadingBackend with 15 concurrent workers.
[Parallel(n_jobs=15)]: Done 2 tasks | elapsed: 1.3min
[Parallel(n_jobs=15)]: Done 98 tasks | elapsed: 7.6min
[Parallel(n_jobs=15)]: Done 258 tasks | elapsed: 18.4min
[Parallel(n_jobs=15)]: Done 482 tasks | elapsed: 34.7min
[Parallel(n_jobs=15)]: Done 770 tasks | elapsed: 55.1min
[Parallel(n_jobs=15)]: Done 1122 tasks | elapsed: 79.9min
[Parallel(n_jobs=15)]: Done 1538 tasks | elapsed: 109.3min
[Parallel(n_jobs=15)]: Done 2018 tasks | elapsed: 142.5min
[Parallel(n_jobs=15)]: Done 2562 tasks | elapsed: 181.0min
[Parallel(n_jobs=15)]: Done 3170 tasks | elapsed: 223.2min
[Parallel(n_jobs=15)]: Done 3842 tasks | elapsed: 271.5min
[Parallel(n_jobs=15)]: Done 4578 tasks | elapsed: 323.2min
[Parallel(n_jobs=15)]: Done 5378 tasks | elapsed: 379.1min
[Parallel(n_jobs=15)]: Done 6242 tasks | elapsed: 440.0min
...
PS Others seem to be experiencing this as well.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the DROrthoForest evaluation entry point and reproduce the reported configuration: 100K rows, about 600 variables, max_depth 10, n_trees 500, and n_jobs 15. Compare evaluation behavior with the related issue #562; done means determining whether the runtime is expected and documenting a supported tuning path or a reproducible performance defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100