huggingface / huggingface/candle

No lazy/mmap loading for GGUF weights (MmapedSafetensors equivalent)

Open
#3,708 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Finding from embedding candle's quantized GGUF models in a long-running server process, verified against the 0.11 sources.

`gguf_file::TensorInfo::read` does `reader.read_exact(&mut raw_data)` per tensor — loading a GGUF always copies the full file into RAM (`from_gguf` → `Content::tensor` → `TensorInfo::read`, candle-core `quantized/gguf_file.rs`). safetensors has `MmapedSafetensors` for lazy loading, but there is no GGUF equivalent, so wrapping the reader over `memmap2` only changes the copy mechanism — it cannot avoid the copy.

An mmap-backed `QTensor` storage for GGUF would:

- cut cold-start load time (measured: ~6s for a 0.5B Q4_K_M, ~16-19s for a 3B on NVMe — essentially all of it read+copy);
- let the OS page cache share weights across processes running the same model;
- reduce peak RSS during load (today the copy briefly doubles resident memory pressure on small machines).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-core/quantized/gguf_file.rs and trace the mentioned from_gguf → Content::tensor → TensorInfo::read path. Compare the existing safetensors MmapedSafetensors approach, then investigate how an mmap-backed QTensor storage could avoid copying each tensor. Done means GGUF weights load lazily without copying the full file into RAM.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.