AlexsJones / AlexsJones/llmfit
Server mode: capacity planning / density calculation for 27B models
- Ngôn ngữ chính
- Rust
- Star
- 36.3k
- Fork
- 2.3k
- Merge trung bình
- 2 ngày 18 giờ
- Pull request đã merge (30 ngày)
- 85
Mô tả
We need to add a **server mode** to llmfit that can compute model capacity (density) given hardware constraints.
**Goal**: Given a set of hardware parameters (RAM size, CPU speed, storage capacity, compute capability), calculate how many 27B models can be run simultaneously and suggest the optimal model type (e.g., GGUF quantization level, ONNX, or llama.cpp). The calculation should consider:
1. Model memory footprint (weights, KV cache) at different quantization levels (fp16, q4, q5, q8, GGUF, ONNX).
2. Per‑GPU/CPU compute throughput (tokens/s) needed for a given inference load.
3. Storage I/O bandwidth for loading models.
4. Overhead for the llmfit runtime (metadata, cache, OS).
5. Optionally, hardware‑specific constraints (e.g., VRAM on GPU, shared memory on CPU, NPU limits).
**Proposed implementation steps**:
- Add a new CLI sub‑command `llmfit density` that accepts flags: `--ram`, `--cpu`, `--gpu`, `--storage`, `--speed`, `--model-size=27b`.
- Internally maintain a table of model size per quantization (e.g., 27B fp16 ≈ 54 GiB, q4_0 ≈ 13.5 GiB, q5_1 ≈ 17 GiB, GGUF ≈ 10‑12 GiB).
- Compute how many copies fit in RAM/VRAM after subtracting a safety margin (e.g., 10%).
- Estimate compute headroom using known token‑per‑second benchmarks for each backend (llama.cpp, ONNX Runtime, GGUF).
- Produce a JSON report listing possible configurations and the maximum concurrent instances.
**Benefits**:
- Gives users a concrete planning tool for deployment sizing.
- Aligns with the requested “server mode” that can make adjusted calculations based on constraints.
- Provides a foundation for future auto‑scaling features.
**Next actions**:
- Draft the data table for model footprints (source: HuggingFace model card stats).
- Implement the CLI stub and unit tests.
- Add documentation to `README.md` under a new “Server mode – capacity planning” section.
Please review and add any additional constraints or desired output formats.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.