Running DeepSeek R1 on 8x RTX 6000 PRO
@pamelap-nvidia is already working on this.
Since Jul 8, 2025.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Hi, is this planned to be supported? With a total of 768GB VRAM, this setup should be able to load the original FP8 weights, and with 14 TB/s combined bandwidth it should also be very fast for low batch sizes in tensor parallel mode!
However, trying to load the model with the sample script, the following error is currently returned:
Unsupported SM version for FP8 block scaling GEMM
The FP4 model does load, but it is vastly slower than expected (only 27 t/s for batch 1, I would have expected an order of magnitude faster for 37B active parameters)
Maybe RTX 6000 PRO support is still work in progress in general? I am seeing a bunch of sm120 related commits recently.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.