[jvm-package] Is there a way to train multiple xgboost models in parallel?
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Use case: a large dataset contains hundreds of users' data. Each user has millions of records showing their daily activities. For each user, I need to train a XGBoost to predict how this particular will behave on an incoming request.
So I have done the following operations:
`
...
val userIdList = rawData.select("userId").distinct
userIdList.foreach(
userId => {
val userData = rawData.filter(rawData("userId") === userId)
val model = new XGBoostClassifier(params)
.....
)
`
However, the above code always generated various types of errors. Some of them were hard to track, just saying "XGBoost trained failed". Some of them were caused by "missing shuffle location ...”
I am wondering if this parallel training is problematic? There are hundreds of users, so it'd be not feasible to submit a spark application for each user and do the training. Is there any other way?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.