huggingface / huggingface/candle
No lazy/mmap loading for GGUF weights (MmapedSafetensors equivalent)
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Finding from embedding candle's quantized GGUF models in a long-running server process, verified against the 0.11 sources.
`gguf_file::TensorInfo::read` does `reader.read_exact(&mut raw_data)` per tensor — loading a GGUF always copies the full file into RAM (`from_gguf` → `Content::tensor` → `TensorInfo::read`, candle-core `quantized/gguf_file.rs`). safetensors has `MmapedSafetensors` for lazy loading, but there is no GGUF equivalent, so wrapping the reader over `memmap2` only changes the copy mechanism — it cannot avoid the copy.
An mmap-backed `QTensor` storage for GGUF would:
- cut cold-start load time (measured: ~6s for a 0.5B Q4_K_M, ~16-19s for a 3B on NVMe — essentially all of it read+copy);
- let the OS page cache share weights across processes running the same model;
- reduce peak RSS during load (today the copy briefly doubles resident memory pressure on small machines).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in candle-core/quantized/gguf_file.rs and trace the mentioned from_gguf → Content::tensor → TensorInfo::read path. Compare the existing safetensors MmapedSafetensors approach, then investigate how an mmap-backed QTensor storage could avoid copying each tensor. Done means GGUF weights load lazily without copying the full file into RAM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100