[RFC] Reduce per-call overhead of CPU inplace prediction (config parse, model-view rebuild)
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Following up on #12306 and #12307 (single-row CPU inplace-predict overhead): after both fixes, I re-profiled the same workload (single row, 127 features, 50 trees, `nthread=1`, `XGBoosterPredictFromDense` in a loop via a proxy DMatrix, Apple M3 Pro, ~4.5us/call) to see what remains. The call is still dominated by per-call setup rather than tree traversal:
| Cost | ~share of call | Notes |
|---|---|---|
| `Json::Load` of the **config** string | ~17% | config bytes are typically identical call-to-call |
| `Json::Load` of the **array-interface** string (inside `ArrayAdapter`) | ~16% | only the data pointer/shape change between calls |
| `HostModel` / `GBTreeModelView` construction | ~19% | rebuilt per call: per-tree view variants, allocations, memcpys — model unchanged between calls |
| Actual prediction work (`FVecFill` + traversal) | ~15% | |
| `SetArray` remainder, output handling, misc | rest | |
So roughly half of a single-row inplace predict is recomputing state that is invariant across calls in a serving loop. Three directions, in decreasing value/effort ratio:
1. **Cache the constructed model view** (`HostModel`) in `PredictionCacheEntry` (or on `GBTree`), invalidated when trees are added or `tree_begin`/`tree_end` differ. This also benefits the DMatrix predict path.
2. **Memoize the parsed prediction config**: if the config bytes equal the previous call's (per booster, thread-local), reuse the parsed `Json`. Alternatively a "prepared predict" handle in the C API, though that's a bigger surface change.
3. **Lighter array-interface parsing** for the adapter path (the document schema is small and fixed; a specialized parser or a `Json::Load` fast path could cut it substantially).
Before writing any code: is there maintainer appetite for work in this direction, and if so, which approach would you prefer for (1) and (2)? Happy to implement and benchmark — same methodology as the two merged PRs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the XGBoosterPredictFromDense call path and the per-call setup identified in the profiling notes. Read PredictionCacheEntry, GBTree, HostModel, and GBTreeModelView to evaluate the proposed cache invalidation boundaries, then compare config and array-interface parsing costs. Done means a maintainer-selected approach is implemented and benchmarked against the stated single-row workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100