kvcache-ai / kvcache-ai/ktransformers

Discussion on the cost-effectiveness of AMX and FP8

Open
#2,004 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

Intel 6416h with AMX
4090 48G

### Reproduction

Although I've been using ktransformers for a long time, I've always had a question: when using a CPU that supports the AMX instruction set, is a GPU of the 40 series or higher with FP8 completely unnecessary? Can I just use `cpu_convert` and `kt-method=AMXINT4` to run all mainstream models, including MiniMax, DeepSeek V3.2, Kimi, and GML 5? In this case, would only 1-2 3090 24G GPUs be sufficient?

However, when I don't use `cpu_convert`, I must use a GPU of the 40 series or higher, but in this case, I still need more memory than the model weights. So what are the advantages of this approach? My actual tests show no significant performance improvement because memory frequency is the bottleneck. But keep in mind that one 4090 48G GPU can buy five 3090 24G GPUs. Is this solution too inefficient?

### Others

_No response_

Contributor guide

Open the contributing guide

Research direction

No file, test, or entry point is identified. Start by locating the `cpu_convert` and `kt-method=AMXINT4` paths, then compare the AMXINT4 and FP8 configurations on the listed hardware and mainstream models; the discussion is complete when the performance and memory trade-offs are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.