Enable Cache Reuse for ExtMemQuantileDMatrix Across Sessions
- Dominant language
- C++
- Stars
- 28.8k
- Forks
- 8.9k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 54
Description
Currently, `ExtMemQuantileDMatrix` creates cache files with names that include the memory address of the `DMatrix` object (e.g., cache-0x7f8b4c000000.gradient_index.page). This makes it impossible to reuse these cache files across different process sessions, even when working with the exact same data.
Although I understand that this design feature was on purpose, it would be good if we can enable something similar as it would save a large amount of pre-computing the same data if we could reuse the `.page` files generated, as long as we know they are being generated from the same input DataIter.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing ExtMemQuantileDMatrix cache-file naming and the DataIter input lifecycle. Determine how cache files can safely identify identical input across process sessions; done means .page files are reused only when they came from the same input, with coverage for cross-session reuse.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100