AlexsJones / AlexsJones/llmfit

Server mode: capacity planning / density calculation for 27B models

Đang mở
#542 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement Feasibility
Ngôn ngữ chính
Rust
Star
36.3k
Fork
2.3k
Merge trung bình
2 ngày 18 giờ
Pull request đã merge (30 ngày)
85

Mô tả

We need to add a **server mode** to llmfit that can compute model capacity (density) given hardware constraints.

**Goal**: Given a set of hardware parameters (RAM size, CPU speed, storage capacity, compute capability), calculate how many 27B models can be run simultaneously and suggest the optimal model type (e.g., GGUF quantization level, ONNX, or llama.cpp). The calculation should consider:
1. Model memory footprint (weights, KV cache) at different quantization levels (fp16, q4, q5, q8, GGUF, ONNX).
2. Per‑GPU/CPU compute throughput (tokens/s) needed for a given inference load.
3. Storage I/O bandwidth for loading models.
4. Overhead for the llmfit runtime (metadata, cache, OS).
5. Optionally, hardware‑specific constraints (e.g., VRAM on GPU, shared memory on CPU, NPU limits).

**Proposed implementation steps**:
- Add a new CLI sub‑command `llmfit density` that accepts flags: `--ram`, `--cpu`, `--gpu`, `--storage`, `--speed`, `--model-size=27b`.
- Internally maintain a table of model size per quantization (e.g., 27B fp16 ≈ 54 GiB, q4_0 ≈ 13.5 GiB, q5_1 ≈ 17 GiB, GGUF ≈ 10‑12 GiB).
- Compute how many copies fit in RAM/VRAM after subtracting a safety margin (e.g., 10%).
- Estimate compute headroom using known token‑per‑second benchmarks for each backend (llama.cpp, ONNX Runtime, GGUF).
- Produce a JSON report listing possible configurations and the maximum concurrent instances.

**Benefits**:
- Gives users a concrete planning tool for deployment sizing.
- Aligns with the requested “server mode” that can make adjusted calculations based on constraints.
- Provides a foundation for future auto‑scaling features.

**Next actions**:
- Draft the data table for model footprints (source: HuggingFace model card stats).
- Implement the CLI stub and unit tests.
- Add documentation to `README.md` under a new “Server mode – capacity planning” section.

Please review and add any additional constraints or desired output formats.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.