CausalLM ERROR: byte not found in vocab

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

Run model CausalLM without errors
https://huggingface.co/TheBloke/CausalLM-14B-GGUF/tree/main

# Current Behavior

Error when loading model (llama-cpp-python installed throught pip, not from source)
```
...
llama_model_loader: - type f32: 81 tensors
llama_model_loader: - type q5_1: 281 tensors
llama_model_loader: - type q6_K: 1 tensors
ERROR: byte not found in vocab: '
'
fish: Job 1, 'python server.py --api --listen…' terminated by signal SIGSEGV (Address boundary error)
```

https://github.com/ggerganov/llama.cpp/issues/3732

# Environment and Context

Docker container with latest

# Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

# Steps to Reproduce

# Failure Logs