Support --ssd-streaming alongside --mtp / --dspark
- Ngôn ngữ chính
- C
- Star
- 22.4k
- Fork
- 2.1k
- Merge trung bình
- 2 ngày 13 giờ
- Pull request đã merge (30 ngày)
- 5
Mô tả
### Feature Description
Currently, attempting to run `ds4` with both `--ssd-streaming` and `--mtp` (or `--dspark`) results in the following operational limitation/error:
> `ds4: --ssd-streaming is not compatible with --mtp yet`
Support for combining `--ssd-streaming` with `--mtp`/DSpark speculative decoding would be extremely beneficial for hardware configurations with memory constraints (such as Apple Silicon devices with 64 GB Unified Memory).
### Motivation & Use Case
On systems like a Mac Studio M1 Ultra (64 GB RAM), running large quantizations requires SSD streaming to keep the memory footprint manageable and prevent OOM / system swapping. However, streaming experts from SSD naturally introduces latency during token generation.
Combining `--ssd-streaming` with DSpark / MTP speculative execution would offer the best of both worlds:
1. Low RAM usage via dynamic expert streaming from fast NVMe SSDs.
2. Improved tokens-per-second generation speed via speculative decoding.
### System Configuration
- **Hardware:** Mac Studio M1 Ultra (64 GB RAM)
- **Tooling:** `ds4` with DeepSeek / DSpark models
Thank you for all the incredible work on `ds4`!
Hướng dẫn đóng góp
Hướng nghiên cứu
Start by locating ds4's argument handling and the compatibility check that rejects --ssd-streaming with --mtp or --dspark. Trace the existing streaming and speculative-decoding paths, then verify that the combined invocation works for the described DeepSeek/DSpark setup without the reported limitation or excessive memory use.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- c
- Lĩnh vực
- ai, performance
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 35/100