dask / dask/dask-ml

Feature request: faster GridsearchCV with XGBoost

Open
#605 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

XGBoost constructs a DMatrix object from the input X, y data it
receives. On its first time calling `train` with this DMatrix, it
caches statistics (histograms) about each column in the DMatrix. For
wide datasets, computing these stats can be very
time-consuming. Running one example grid search with a 1000-column
matrix (x 100k rows), using GridsearchCV was 2.8x longer than building
a DMatrix once manually for each fold and calling xgboost.train in a
loop. Profiling shows that the time goes to both DMatrix building and
stats computation.

GridsearchCV CV caching essentially works with pandas or numpy-style
objects with separate X and y. DMatrix includes both X and y in the
same object, so it needs a slight hack to fit into the current framework.

One option is to allow users to supply a custom replacement for the
`_extract` method in methods.py. For XGBoost, a user would supply something like:

```
def _extract_dmatrix(self, X, y, n, is_x=True, is_train=True):
if self.cache is not None and (n, is_x, is_train) in self.cache:
return self.cache[n, is_x, is_train]
if not is_x:
return None

inds = self.splits[n][0] if is_train else self.splits[n][1]
x_part = _safe_indexing(X, inds)
y_part = _safe_indexing(y, inds)

import xgboost as xgb
# TODO: in practice, there may be additional params like weights
result = xgb.DMatrix(x_part, y_part)
print("Converted to dmatrix for cache with _extract_dmatrix")

if self.cache is not None:
self.cache[n, is_x, is_train] = result
return result
```

This returns None for the y and a DMatrix with X and y with is_x, so
it is abusing the interface a bit, though it seem to work. Note that
this also requires a custom XGBoost wrapper with a `fit` method
accepting a DMatrix, but that is very short.

Thoughts? Better ways to do this and cache DMatrix objects? Thanks!
This relates to Issue #443, which is also looking for advice on XGBoost gridsearch.

Tagging @trivialfis for possible better ideas...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.