Fine tuned model does not work (GGUF, q4_k medium)

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

High level API inference via [example script](https://github.com/abetlen/llama-cpp-python/blob/main/examples/high_level_api/high_level_api_inference.py).

Langchain RAG.

# Current Behavior

Model cannot generate responses. Example output:
{
"id": "cmpl-7b450897-1133-4af3-bc5d-6d1582979b69",
"object": "text_completion",
"created": 1711633841,
"model": "./models/LLaMA-2-7B-32K-Q4_K_M.gguf",
"choices": [
{
"text": "Q: Name the planets in the solar system? A: 9.",
"index": 0,
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 15,
"completion_tokens": 3,
"total_tokens": 18
}
}

For other questions, there isn't any answer.
Other models worked as expected.

For RAG, model just responded with ''''''''''
Other models worked as expected.

# Environment and Context

I have converted [LLaMA2 7B 32k model](https://huggingface.co/togethercomputer/LLaMA-2-7B-32K) to GGUF format and then quantized it to q4. I have also done this with original Llama model from Meta and it works just fine. I am using the latest libraries.

* Physical (or virtual) hardware you are using, e.g. for Linux:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
CPU family: 6
Model: 158
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
Stepping: 10
BogoMIPS: 7391.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopolog
y cpuid pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp
tpr_shadow vnmi ept vpid ept_ad fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves md_clear flush_l1d arch_capabilities

* Operating System, e.g. for Linux:

Windows 11, with WSL2 Ubuntu

* SDK version, e.g. for Linux:

Python 3.10.12
Llama-cpp-python is latest version

# Steps to Reproduce

1. download [model](https://huggingface.co/rozek/LLaMA-2-7B-32K_GGUF)
2. run with [example script](https://github.com/abetlen/llama-cpp-python/blob/main/examples/high_level_api/high_level_api_inference.py) or Langchain
3. I run python scripts via CLI