dmlc / dmlc/xgboost

[RFC] Reduce per-call overhead of CPU inplace prediction (config parse, model-view rebuild)

Open
#12,311 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Following up on #12306 and #12307 (single-row CPU inplace-predict overhead): after both fixes, I re-profiled the same workload (single row, 127 features, 50 trees, `nthread=1`, `XGBoosterPredictFromDense` in a loop via a proxy DMatrix, Apple M3 Pro, ~4.5us/call) to see what remains. The call is still dominated by per-call setup rather than tree traversal:

| Cost | ~share of call | Notes |
|---|---|---|
| `Json::Load` of the **config** string | ~17% | config bytes are typically identical call-to-call |
| `Json::Load` of the **array-interface** string (inside `ArrayAdapter`) | ~16% | only the data pointer/shape change between calls |
| `HostModel` / `GBTreeModelView` construction | ~19% | rebuilt per call: per-tree view variants, allocations, memcpys — model unchanged between calls |
| Actual prediction work (`FVecFill` + traversal) | ~15% | |
| `SetArray` remainder, output handling, misc | rest | |

So roughly half of a single-row inplace predict is recomputing state that is invariant across calls in a serving loop. Three directions, in decreasing value/effort ratio:

1. **Cache the constructed model view** (`HostModel`) in `PredictionCacheEntry` (or on `GBTree`), invalidated when trees are added or `tree_begin`/`tree_end` differ. This also benefits the DMatrix predict path.
2. **Memoize the parsed prediction config**: if the config bytes equal the previous call's (per booster, thread-local), reuse the parsed `Json`. Alternatively a "prepared predict" handle in the C API, though that's a bigger surface change.
3. **Lighter array-interface parsing** for the adapter path (the document schema is small and fixed; a specialized parser or a `Json::Load` fast path could cut it substantially).

Before writing any code: is there maintainer appetite for work in this direction, and if so, which approach would you prefer for (1) and (2)? Happy to implement and benchmark — same methodology as the two merged PRs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the XGBoosterPredictFromDense call path and the per-call setup identified in the profiling notes. Read PredictionCacheEntry, GBTree, HostModel, and GBTreeModelView to evaluate the proposed cache invalidation boundaries, then compare config and array-interface parsing costs. Done means a maintainer-selected approach is implemented and benchmarked against the stated single-row workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.