Model Initialisation happening per request - JAVA GPU inferencing
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
We are observing high latency when running real-time batch inference on GPUs using the xgboost4j-gpu Java library. Our investigation suggests that a model initialization step may be occurring on every predict call, even when Booster objects are reused.
Please see the attached flame graph, which highlights the time spent in the model initialization phase during a predict call.
Code reference we are using for prediction.
```
try {
DMatrix dmat = new DMatrix(features, nRows, nCols, Float.NaN);
booster = boosters.borrowObject(); // taking a pre initialised Booster object from a pool
Timer.Context context = MetricUtils.addTimer("xgb_true_time", this.getClass()).time();
float[][] modelScores = booster.predict(dmat);
context.stop();
} catch (XGBoostError e) {
e.printStackTrace();
} finally {
if (booster != null ) {
boosters.returnObject(booster); // returning back to pool
}
```
**Question**
Is this internal model initialization for each batch prediction an expected behavior for the GPU predictor, or could this indicate a potential bug or a misconfiguration on our part? Any guidance on how to avoid this per-batch setup cost would be greatly appreciated.
(I was referring to this section, and from the source code, I also understood that Init is called for every predict call , https://github.com/dmlc/xgboost/blob/release_2.0.0/src/predictor/gpu_predictor.cu#L890)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked src/predictor/gpu_predictor.cu section around line 890 and the Java prediction code using booster.predict(dmat). Trace whether Init runs for each prediction despite reusing Booster objects, and check the attached flame graph context. Done means determining whether this is expected GPU behavior, a misconfiguration, or a reproducible defect with actionable guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100