abetlen / abetlen/llama-cpp-python
CUDA: Generate error message for unsupported quantizations like iq4_nl
- 主要語言
- Python
- 星號
- 10.6k
- 分支
- 1.4k
- PR 合併指標
- PR 指標待擷取
描述
# The Problem
`llama.cpp` crashes instead of reporting that it does not support iq4_nl quantization.
`./llama-cli -ngl 1 -m ~/.local/share/models/Phi-3-mini-4k-instruct-IQ4_NL.gguf`
available from https://huggingface.co/lmstudio-community/Phi-3-mini-4k-instruct-GGUF/tree/main
# Expected results
"Example: IQ4_NL does not support -ngl. Please run without -ngl flag"
# Current Behavior
```
...
Aborted (core dumped)
```
# The problem
Toward the end of ggml-cuda/dmmv.cu:665, it aborts on this assertion.
```
case GGML_TYPE_Q6_K:
dequantize_mul_mat_vec_q6_K_cuda(src0_dd_i, src1_ddf_i, dst_dd_i, ne00,
row_diff, stream);
break;
case GGML_TYPE_F16:
convert_mul_mat_vec_f16_cuda(src0_dd_i, src1_dfloat, dst_dd_i, ne00,
row_diff, stream);
break;
default:
-------> GGML_ASSERT(false);
break;
}
```
`$ lscpu`
[cpu.txt](https://github.com/abetlen/llama-cpp-python/files/15381753/cpu.txt)
`lspci`
...
```
01:00.0 VGA compatible controller: NVIDIA Corporation GM204GLM [Quadro M3000M] (rev a1) (prog-if 00 [VGA controller])
DeviceName: 0
Subsystem: Hewlett-Packard Company Device 1630
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
```
`$ uname -a`
Linux fedora 6.8.9-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Thu May 2 18:44:19 UTC 2024 x86_64 GNU/Linux
貢獻指南
評估
這個 Issue 還沒有評估資料。