Feature Request: On-disk (out-of-core) streaming of model weights for low-RAM inference

Summary

Please consider adding support for on-disk / out-of-core streaming of model weights in MLX — e.g., via memory-mapping and block-wise loading.
The goal is to enable running LLMs much larger than system RAM by keeping only the active working set in memory.

Motivation

Many Mac users run MLX on devices with 8–16 GB shared RAM, which makes it impossible to load larger models even when quantized.
With upcoming high-bandwidth storage (PCIe 5.0/6.0 SSDs reaching 10–20 GB/s), it becomes feasible to stream most weights directly from disk instead of loading them fully into memory.

This would allow models such as Qwen3-80B- next (3-bit active ) to run locally with ~3–4 GB active RAM, keeping the remaining weights on SSD and streamed on demand.

What’s already known to work
• llama.cpp and other GGML projects use mmap to avoid copying weights into RAM.
• vLLM demonstrated the viability of paged / out-of-core techniques with an IO-aware scheduler and block-level attention caching.
• PyTorch has memory-mapped tensor support as well.

These examples show that the technique is feasible and useful when implemented with an appropriate chunked weight format.

Requested capabilities
1. Memory-mapped model weights
Load .mlx tensor files via mmap, so weights do not need to be copied into RAM.
2. Block-wise / chunked tensor storage
Store weights in aligned blocks so the runtime can request only the needed parts of each layer.
3. Asynchronous prefetching & residency policy
A simple IO scheduler that preloads blocks for the next N layers/tokens and evicts old ones (LRU or similar).
4. Optional API flag, e.g.:

model = mlx.load("model.mlx", streaming=True, mmap=True)

Benefits for MLX
• Run LLMs far larger than device RAM.
• Drastically lower memory pressure on machines with 8–16 GB.
• Faster startup (no full weight copy).
• Makes MLX attractive for edge/desktop inference as PCIe bandwidth grows.

Why this fits MLX well

MLX already has a clean tensor abstraction and efficient kernels, so adding a streaming backend would make it a uniquely powerful local inference engine for macOS.
This is especially relevant as storage bandwidth is growing faster than RAM capacity.

Closing

If implemented, this feature could also serve as a way to measure real bottlenecks — IO latency, block size behavior, DMA transfer limits — and provide insights that may inform future hardware optimization and model-storage formats.

The issue names no source files or tests; begin by locating the model-loading and tensor-storage paths behind the proposed mlx.load(..., streaming=True, mmap=True) API. Define the scope and acceptance criteria for memory mapping, chunked weights, prefetching, and eviction before implementation can begin.