abetlen / abetlen/llama-cpp-python

CUDA: Generate error message for unsupported quantizations like iq4_nl

Abierto
#1,473 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

# The Problem
`llama.cpp` crashes instead of reporting that it does not support iq4_nl quantization.

`./llama-cli -ngl 1 -m ~/.local/share/models/Phi-3-mini-4k-instruct-IQ4_NL.gguf`
available from https://huggingface.co/lmstudio-community/Phi-3-mini-4k-instruct-GGUF/tree/main

# Expected results

"Example: IQ4_NL does not support -ngl. Please run without -ngl flag"

# Current Behavior
```
...
Aborted (core dumped)
```

# The problem

Toward the end of ggml-cuda/dmmv.cu:665, it aborts on this assertion.

```
case GGML_TYPE_Q6_K:
dequantize_mul_mat_vec_q6_K_cuda(src0_dd_i, src1_ddf_i, dst_dd_i, ne00,
row_diff, stream);
break;
case GGML_TYPE_F16:
convert_mul_mat_vec_f16_cuda(src0_dd_i, src1_dfloat, dst_dd_i, ne00,
row_diff, stream);
break;
default:
-------> GGML_ASSERT(false);
break;
}
```

`$ lscpu`
[cpu.txt](https://github.com/abetlen/llama-cpp-python/files/15381753/cpu.txt)

`lspci`
...
```
01:00.0 VGA compatible controller: NVIDIA Corporation GM204GLM [Quadro M3000M] (rev a1) (prog-if 00 [VGA controller])
DeviceName: 0
Subsystem: Hewlett-Packard Company Device 1630
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
```
`$ uname -a`
Linux fedora 6.8.9-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 2 18:44:19 UTC 2024 x86_64 GNU/Linux

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.