dmlc / dmlc/xgboost

Enable Cache Reuse for ExtMemQuantileDMatrix Across Sessions

Open
#11,728 4 comments 0 reactions 0 assignees View on GitHub
external-memory feature-request
Dominant language
C++
Stars
28.8k
Forks
8.9k
Avg merge
1d 12h
Merged PRs (30d)
54

Description

Currently, `ExtMemQuantileDMatrix` creates cache files with names that include the memory address of the `DMatrix` object (e.g., cache-0x7f8b4c000000.gradient_index.page). This makes it impossible to reuse these cache files across different process sessions, even when working with the exact same data.

Although I understand that this design feature was on purpose, it would be good if we can enable something similar as it would save a large amount of pre-computing the same data if we could reuse the `.page` files generated, as long as we know they are being generated from the same input DataIter.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing ExtMemQuantileDMatrix cache-file naming and the DataIter input lifecycle. Determine how cache files can safely identify identical input across process sessions; done means .page files are reused only when they came from the same input, with coverage for cross-session reuse.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.