Feature request: faster GridsearchCV with XGBoost
- Dominant language
- Python
- Stars
- 951
- Forks
- 262
- PR merge metrics
- No merged PRs in 30d
Description
XGBoost constructs a DMatrix object from the input X, y data it
receives. On its first time calling `train` with this DMatrix, it
caches statistics (histograms) about each column in the DMatrix. For
wide datasets, computing these stats can be very
time-consuming. Running one example grid search with a 1000-column
matrix (x 100k rows), using GridsearchCV was 2.8x longer than building
a DMatrix once manually for each fold and calling xgboost.train in a
loop. Profiling shows that the time goes to both DMatrix building and
stats computation.
GridsearchCV CV caching essentially works with pandas or numpy-style
objects with separate X and y. DMatrix includes both X and y in the
same object, so it needs a slight hack to fit into the current framework.
One option is to allow users to supply a custom replacement for the
`_extract` method in methods.py. For XGBoost, a user would supply something like:
```
def _extract_dmatrix(self, X, y, n, is_x=True, is_train=True):
if self.cache is not None and (n, is_x, is_train) in self.cache:
return self.cache[n, is_x, is_train]
if not is_x:
return None
inds = self.splits[n][0] if is_train else self.splits[n][1]
x_part = _safe_indexing(X, inds)
y_part = _safe_indexing(y, inds)
import xgboost as xgb
# TODO: in practice, there may be additional params like weights
result = xgb.DMatrix(x_part, y_part)
print("Converted to dmatrix for cache with _extract_dmatrix")
if self.cache is not None:
self.cache[n, is_x, is_train] = result
return result
```
This returns None for the y and a DMatrix with X and y with is_x, so
it is abusing the interface a bit, though it seem to work. Note that
this also requires a custom XGBoost wrapper with a `fit` method
accepting a DMatrix, but that is very short.
Thoughts? Better ways to do this and cache DMatrix objects? Thanks!
This relates to Issue #443, which is also looking for advice on XGBoost gridsearch.
Tagging @trivialfis for possible better ideas...
Contributor guide
Assessment
This issue has not been assessed yet.