kvcache-ai / kvcache-ai/ktransformers
Discussion on the cost-effectiveness of AMX and FP8
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
Intel 6416h with AMX
4090 48G
### Reproduction
Although I've been using ktransformers for a long time, I've always had a question: when using a CPU that supports the AMX instruction set, is a GPU of the 40 series or higher with FP8 completely unnecessary? Can I just use `cpu_convert` and `kt-method=AMXINT4` to run all mainstream models, including MiniMax, DeepSeek V3.2, Kimi, and GML 5? In this case, would only 1-2 3090 24G GPUs be sufficient?
However, when I don't use `cpu_convert`, I must use a GPU of the 40 series or higher, but in this case, I still need more memory than the model weights. So what are the advantages of this approach? My actual tests show no significant performance improvement because memory frequency is the bottleneck. But keep in mind that one 4090 48G GPU can buy five 3090 24G GPUs. Is this solution too inefficient?
### Others
_No response_
Contributor guide
Research direction
No file, test, or entry point is identified. Start by locating the `cpu_convert` and `kt-method=AMXINT4` paths, then compare the AMXINT4 and FP8 configurations on the listed hardware and mainstream models; the discussion is complete when the performance and memory trade-offs are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100