dmlc / dmlc/xgboost

[jvm-package] Is there a way to train multiple xgboost models in parallel?

Open
#4,004 5 comments 0 reactions 0 assignees View on GitHub
type: java-scala
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Use case: a large dataset contains hundreds of users' data. Each user has millions of records showing their daily activities. For each user, I need to train a XGBoost to predict how this particular will behave on an incoming request.
So I have done the following operations:
`
...
val userIdList = rawData.select("userId").distinct

userIdList.foreach(
userId => {
val userData = rawData.filter(rawData("userId") === userId)
val model = new XGBoostClassifier(params)
.....
)
`
However, the above code always generated various types of errors. Some of them were hard to track, just saying "XGBoost trained failed". Some of them were caused by "missing shuffle location ...”

I am wondering if this parallel training is problematic? There are hundreds of users, so it'd be not feasible to submit a spark application for each user and do the training. Is there any other way?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.