dmlc / dmlc/xgboost

[jvm-packages] training multi xgboost models using ThreadPool occurs error

Open
#7,234 2 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

# background
I have multi datasets with same schema, so I want to train multi models in parallel. I call scala `Executors.newFixedThreadPool` api to do this. But `Doing rabit call after Finalize` error happend in `first` stage
![image](https://user-images.githubusercontent.com/21337200/133548124-d1af9ed8-0e19-45f5-b2ed-86e65d4323dd.png)

## enviroment
- spark: 2.2
- xgboost: 0.82

# what I have try
i follow the [closed pr](https://github.com/ebernhardson/xgboost/commit/02e2714d5b0d0605bd1f0d81921599129741ce63) in this [issue](https://github.com/dmlc/xgboost/issues/2449) to return `Array[Byte]` rather than `Booster`。Now, it looks like the application can run successfully

# problem
I want to know, why `Doing rabit call after Finalize` error occurs when training models in parallel and why the solution mentioned above works?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the JVM-package training path that uses Scala Executors.newFixedThreadPool, then compare the Booster return behavior with the cited closed commit that returns Array[Byte]. Reproduce the parallel training case on Spark 2.2 with xgboost 0.82 and trace when the Rabit call occurs relative to finalization. Done means explaining the error and why the cited change avoids it.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.