llama-cpp-python crashing on CUBLAS and CLBLAST with std::length_error

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [X] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [X] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [X] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [X] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

Trying to run this simple script
```
from llama_cpp import Llama
llm = Llama(model_path="./gen-z.gguf", n_gpu_layers=1)

llm("Hello")
```

I expect that script to not crash and have the same result as calling the llama.cpp cli

`./main -m /models/gen-z.gguf -p "hi" -n 512 --n-gpu-layers 1`

# Current Behavior

Using CUBLAS with cuda 11.7 and 12.1 `llm` crashes with
```
terminate called after throwing an instance of 'std::length_error'
what(): basic_string::_M_create
Aborted (core dumped)
```

Using CLBLAST it just segfaults.

# Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

* Physical (or virtual) hardware you are using, e.g. for Linux:

`$ lscpu`
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7900X 12-Core Processor
CPU family: 25
Model: 97
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 1
Stepping: 2
CPU(s) scaling MHz: 80%
CPU max MHz: 5733.0000
CPU min MHz: 400.0000
BogoMIPS: 9385.62
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pg
e mca cmov pat pse36 clflush mmx fxsr sse sse2 ht s
yscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constan
t_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid ex
td_apicid aperfmperf rapl pni pclmulqdq monitor sss
e3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes x
save avx f16c rdrand lahf_lm cmp_legacy svm extapic
cr8_legacy abm sse4a misalignsse 3dnowprefetch osv
w ibs skinit wdt tce topoext perfctr_core perfctr_n
b bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pst
ate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanc
ed vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpci
d cqm rdt_a avx512f avx512dq rdseed adx smap avx512
ifma clflushopt clwb avx512cd sha_ni avx512bw avx51
2vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occu
p_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzer
o irperf xsaveerptr rdpru wbnoinvd cppc arat npt lb
rv svm_lock nrip_save tsc_scale vmcb_clean flushbya
sid decodeassists pausefilter pfthreshold avic v_vm
save_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi
umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq a
vx512_vnni avx512_bitalg avx512_vpopcntdq rdpid ove
rflow_recov succor smca fsrm flush_l1d
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 384 KiB (12 instances)
L1i: 384 KiB (12 instances)
L2: 12 MiB (12 instances)
L3: 64 MiB (2 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-23
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Mitigation; Speculative Store Bypass disabled via p
rctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user poi
nter sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditi
onal, STIBP always-on, RSB filling, PBRSB-eIBRS Not
affected
Srbds: Not affected
Tsx async abort: Not affected
```

* Operating System, e.g. for Linux:

`$ uname -a`
```
Linux darch 6.5.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 13 Sep 2023 08:37:40 +0000 x86_64 GNU/Linux
```

* SDK version, e.g. for Linux:

```
$ python3 --version
$ make --version
$ g++ --version
```
```
Python 3.11.5
```
```
GNU Make 4.4.1
```
```
g++ (GCC) 13.2.1 20230801
```

# Failure Information (for bugs)

This seems to work fine with llama.cpp's own CLI, so it seems like a llama-cpp-python issue.

# Steps to Reproduce
Compile the latest llama.cpp using `make` or their provided docker container.
set the LLAMA_CPP_LIB to the location of the compiled libllama.so
Run the script above.