py-why / py-why/EconML

Tips on tuning DROrthoForest - is its extreme slowness during evaluation expected?

Open
#566 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
4.8k
Forks
827
PR merge metrics
No merged PRs in 30d

Description

Can you please provide some tips on tuning DROrthoForest? From its description it sounds like it would be especially useful on larger datasets due to a local component to the fit, but its performance (in terms of speed) when I tried that is terrible.

I fit it on a dataset with 100K data points and some 600 variables (most of them one-hot-encoded categoricals), and the fit itself is fast enough (about 20 mins), but evaluating the model on that same dataset has so far run 48 hours and shows no signs of completing.

Some tests show that the max_depth and n_trees parameters have almost no effect on speed of evaluation.

First of all, is this intended behavior? It seems weird that it takes orders of magnitude longer to evaluate a forest-style model than to fit it.

Secondly, how can I tweak it so the model becomes usable on datasets of one-two million rows?

Thanks a lot!

fitting backdoor.econml.orf.DROrthoForest {'init_params': {'propensity_model': DummyClassifier(), 'model_Y': Ridge(alpha=0.01), 'n_jobs': 15, 'max_depth': 10, 'n_trees': 500, 'min_leaf_size': 1258, 'backend': 'threading'}, 'fit_params': {}}
[Parallel(n_jobs=15)]: Using backend ThreadingBackend with 15 concurrent workers.
[Parallel(n_jobs=15)]: Done   2 tasks      | elapsed:   19.0s
[Parallel(n_jobs=15)]: Done  98 tasks      | elapsed:  2.2min
[Parallel(n_jobs=15)]: Done 258 tasks      | elapsed:  5.5min
[Parallel(n_jobs=15)]: Done 500 out of 500 | elapsed: 10.4min finished
[Parallel(n_jobs=15)]: Using backend ThreadingBackend with 15 concurrent workers.
[Parallel(n_jobs=15)]: Done   2 tasks      | elapsed:   18.6s
[Parallel(n_jobs=15)]: Done  98 tasks      | elapsed:  2.1min
[Parallel(n_jobs=15)]: Done 258 tasks      | elapsed:  5.5min
[Parallel(n_jobs=15)]: Done 500 out of 500 | elapsed: 10.4min finished
[Parallel(n_jobs=15)]: Using backend ThreadingBackend with 15 concurrent workers.
[Parallel(n_jobs=15)]: Done   2 tasks      | elapsed:  1.3min
[Parallel(n_jobs=15)]: Done  98 tasks      | elapsed:  7.6min
[Parallel(n_jobs=15)]: Done 258 tasks      | elapsed: 18.4min
[Parallel(n_jobs=15)]: Done 482 tasks      | elapsed: 34.7min
[Parallel(n_jobs=15)]: Done 770 tasks      | elapsed: 55.1min
[Parallel(n_jobs=15)]: Done 1122 tasks      | elapsed: 79.9min
[Parallel(n_jobs=15)]: Done 1538 tasks      | elapsed: 109.3min
[Parallel(n_jobs=15)]: Done 2018 tasks      | elapsed: 142.5min
[Parallel(n_jobs=15)]: Done 2562 tasks      | elapsed: 181.0min
[Parallel(n_jobs=15)]: Done 3170 tasks      | elapsed: 223.2min
[Parallel(n_jobs=15)]: Done 3842 tasks      | elapsed: 271.5min
[Parallel(n_jobs=15)]: Done 4578 tasks      | elapsed: 323.2min
[Parallel(n_jobs=15)]: Done 5378 tasks      | elapsed: 379.1min
[Parallel(n_jobs=15)]: Done 6242 tasks      | elapsed: 440.0min
...

PS Others seem to be experiencing this as well.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the DROrthoForest evaluation entry point and reproduce the reported configuration: 100K rows, about 600 variables, max_depth 10, n_trees 500, and n_jobs 15. Compare evaluation behavior with the related issue #562; done means determining whether the runtime is expected and documenting a supported tuning path or a reproducible performance defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.