huggingface / huggingface/candle

Transparent Huge Pages Support

Open
#2,149 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

When working with CPU tensors on Linux, transparent huge pages (THP) can provide a big speedup. For example, I see a 15% increase in speed for my code when I turn it on. However, for many operating systems such as Ubuntu, the default behavior for THP is "madvise", which means that madvise must be called with the proper flags on each memory segment that we want THP for. In NumPy, they turn on THP with madvise for any arrays that are 4 MB or larger, when they're on Linux 4.6+ (initial commit: https://github.com/numpy/numpy/commit/7180479b7ce3e3b6455da66d0679274671a46bdc). It would be great to have similar behavior for candle. I'm not exactly sure how this would be implemented, as CpuStorage takes a Vec that may have already been paged in, but maybe calling madvise somehow for all usages of CpuStorage in cpu_backend.rs would cover the broad strokes.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.