antirez / antirez/ds4

FP4 inference for 4 bit

Aperta
#74 8 commenti 2 reazioni 0 assegnatari Vedi su GitHub
correctness
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

Right now we have Q4_K GGUF, however FP4 is the way the model is shipped and is numerically different than Q4_K which is uniform quantization. Given that llama.cpp is able to do the inference of GPT120B OSS (I believe) fast enough, it should be possible to support at least as *optional format* the FP4 weights, in order to really do the inference of the Real Thing that DeepSeek shipped. I doubt there are large differences as even the 2 bit quants work well, but... still. Not a priority but something to remember it is worth investigating.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.