dmlc / dmlc/xgboost

Model Initialisation happening per request - JAVA GPU inferencing

Open
#11,668 9 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

We are observing high latency when running real-time batch inference on GPUs using the xgboost4j-gpu Java library. Our investigation suggests that a model initialization step may be occurring on every predict call, even when Booster objects are reused.
Please see the attached flame graph, which highlights the time spent in the model initialization phase during a predict call.

Image

Code reference we are using for prediction.

```
try {
DMatrix dmat = new DMatrix(features, nRows, nCols, Float.NaN);
booster = boosters.borrowObject(); // taking a pre initialised Booster object from a pool
Timer.Context context = MetricUtils.addTimer("xgb_true_time", this.getClass()).time();

float[][] modelScores = booster.predict(dmat);
context.stop();

} catch (XGBoostError e) {
e.printStackTrace();
} finally {
if (booster != null ) {
boosters.returnObject(booster); // returning back to pool
}
```

**Question**
Is this internal model initialization for each batch prediction an expected behavior for the GPU predictor, or could this indicate a potential bug or a misconfiguration on our part? Any guidance on how to avoid this per-batch setup cost would be greatly appreciated.

(I was referring to this section, and from the source code, I also understood that Init is called for every predict call , https://github.com/dmlc/xgboost/blob/release_2.0.0/src/predictor/gpu_predictor.cu#L890)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked src/predictor/gpu_predictor.cu section around line 890 and the Java prediction code using booster.predict(dmat). Trace whether Init runs for each prediction despite reusing Booster objects, and check the attached flame graph context. Done means determining whether this is expected GPU behavior, a misconfiguration, or a reproducible defect with actionable guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.