Significant loss of performance from v0.2.28 to v0.2.29 on Mac Metal GPU

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

The performance of my example code below should stay the same for different versions of llama-cpp-python

# Current Behavior

When I update my llama-cpp-python dependency from 0.2.28 to 0.2.29 or higher I experience a significant (>50%) loss of performance in the number of tokens generated per second for my example code below.

From:
```
Version llama_cpp_python 0.2.28
....
llama_print_timings: eval time = 8177.10 ms / 207 runs ( 39.50 ms per token, 25.31 tokens per second)
```
to
```
Version llama_cpp_python 0.2.29
...
llama_print_timings: eval time = 17605.85 ms / 207 runs ( 85.05 ms per token, 11.76 tokens per second)
```

# Environment and Context

I'm running an a mac mini with m2 pro processor
```
$ system_profiler SPSoftwareDataType SPHardwareDataType
Software:
System Software Overview:
System Version: macOS 13.5 (22G74)
Kernel Version: Darwin 22.6.0
...
Hardware:
Hardware Overview:
Model Name: Mac mini
Model Identifier: Mac14,12
Model Number: MNH73D/A
Chip: Apple M2 Pro
Total Number of Cores: 10 (6 performance and 4 efficiency)
Memory: 16 GB
...
```

```
$ uname -a
Darwin Mac-mini.fritz.box 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:21:53 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6020 arm64
$ python3 --version
Python 3.9.6
$ pip list | egrep "langchain"
langchain 0.1.1
langchain-community 0.0.13
langchain-core 0.1.13
```

# Steps to Reproduce

Running the following example code after downloading the `mistral-7b-openorca.Q5_K_M.gguf` model

```
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_community.llms import LlamaCpp

import llama_cpp

print("Version llama_cpp_python", llama_cpp.__version__)

n_gpu_layers = 1 # Metal set to 1 is enough.
n_batch = 2048 # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.
n_ctx = 10240 # Our context size in tokens

llm = LlamaCpp(
model_path="./models/mistral-7b-openorca.Q5_K_M.gguf",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
n_ctx=n_ctx,
temperature=0.0,
f16_kv=True, # MUST set to True, otherwise you will run into problem after a couple of calls
)

template = """Question: {question}

Answer: Let's work this out in a step by step way to be sure we have the right answer."""

prompt = PromptTemplate(template=template, input_variables=["question"])
llm_chain = LLMChain(prompt=prompt, llm=llm)

result = llm_chain.invoke({"question": "Why is the sky blue?"})
print(result)
```

# Failure Logs

For version 0.2.28 this will generate the following output

```
$ python llm_performance_bug.py
Version llama_cpp_python 0.2.28
llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from ./models/mistral-7b-openorca.Q5_K_M.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = open-orca_mistral-7b-openorca
llama_model_loader: - kv 2: llama.context_length u32 = 32768
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 17
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32002] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32002] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32002] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 32000
llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 19: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q5_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 261/32002 ).
llm_load_print_meta: format = GGUF V2
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32002
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q5_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.78 GiB (5.67 BPW)
llm_load_print_meta: general.name = open-orca_mistral-7b-openorca
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 32000 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.11 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size = 4893.72 MiB, ( 4893.78 / 10922.67)
llm_load_tensors: system memory used = 4893.11 MiB
..................................................................................................
llama_new_context_with_model: n_ctx = 10240
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Pro
ggml_metal_init: picking default device: Apple M2 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/andreas/.local/share/virtualenvs/homepage-YjlalDBE/lib/python3.9/site-packages/llama_cpp/ggml-metal.metal'
ggml_metal_init: GPU name: Apple M2 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB
ggml_metal_init: maxTransferRate = built-in GPU
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 1280.00 MiB, ( 6174.47 / 10922.67)
llama_new_context_with_model: KV self size = 1280.00 MiB, K (f16): 640.00 MiB, V (f16): 640.00 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 0.02 MiB, ( 6174.48 / 10922.67)
llama_build_graph: non-view tensors processed: 676/676
llama_new_context_with_model: compute buffer total size = 2739.20 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 2736.02 MiB, ( 8910.48 / 10922.67)
AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |

llama_print_timings: load time = 2696.75 ms
llama_print_timings: sample time = 22.13 ms / 208 runs ( 0.11 ms per token, 9399.01 tokens per second)
llama_print_timings: prompt eval time = 2696.68 ms / 35 tokens ( 77.05 ms per token, 12.98 tokens per second)
llama_print_timings: eval time = 8177.10 ms / 207 runs ( 39.50 ms per token, 25.31 tokens per second)
llama_print_timings: total time = 11158.77 ms
{'question': 'Why is the sky blue?', 'text': "\n\n1. The Earth is surrounded by an atmosphere, which is made up of many gases including nitrogen and oxygen.\n2. When sunlight enters the Earth's atmosphere, it contains all the colors of the rainbow - red, orange, yellow, green, blue, indigo, and violet.\n3. As the sunlight passes through the atmosphere, some of the colors are scattered in different directions by the gas molecules. This is called scattering.\n4. The shorter wavelength colors (blue, indigo, and violet) are scattered more than the longer wavelength colors (red, orange, yellow, and green).\n5. When we look up at the sky, we see the blue color because our eyes are most sensitive to the blue light that has been scattered by the atmosphere.\n6. The other colors of the rainbow are also present in the sunlight, but they are not as easily seen because they are either absorbed by the atmosphere or scattered less than the blue light."}
ggml_metal_free: deallocating
```

For version 0.2.29 it will generate the following output

```
$ python llm_performance_bug.py
Version llama_cpp_python 0.2.29
llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from ./models/mistral-7b-openorca.Q5_K_M.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = open-orca_mistral-7b-openorca
llama_model_loader: - kv 2: llama.context_length u32 = 32768
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096
llama_model_loader: - kv 4: llama.block_count u32 = 32
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: llama.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 11: general.file_type u32 = 17
llama_model_loader: - kv 12: tokenizer.ggml.model str = llama
llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32002] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32002] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32002] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 32000
llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 19: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q5_K: 193 tensors
llama_model_loader: - type q6_K: 33 tensors
llm_load_vocab: special tokens definition check successful ( 261/32002 ).
llm_load_print_meta: format = GGUF V2
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32002
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q5_K - Medium
llm_load_print_meta: model params = 7.24 B
llm_load_print_meta: model size = 4.78 GiB (5.67 BPW)
llm_load_print_meta: general.name = open-orca_mistral-7b-openorca
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 32000 ''
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_tensors: ggml ctx size = 0.22 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size = 151.02 MiB, ( 151.08 / 10922.67)
llm_load_tensors: offloading 1 repeating layers to GPU
llm_load_tensors: offloaded 1/33 layers to GPU
llm_load_tensors: CPU buffer size = 4893.00 MiB
llm_load_tensors: Metal buffer size = 151.00 MiB
...................................................................................................
llama_new_context_with_model: n_ctx = 10240
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Pro
ggml_metal_init: picking default device: Apple M2 Pro
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/andreas/.local/share/virtualenvs/homepage-YjlalDBE/lib/python3.9/site-packages/llama_cpp/ggml-metal.metal'
ggml_metal_init: GPU name: Apple M2 Pro
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction support = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 11453.25 MB
ggml_metal_init: maxTransferRate = built-in GPU
llama_kv_cache_init: CPU KV buffer size = 1280.00 MiB
llama_new_context_with_model: KV self size = 1280.00 MiB, K (f16): 640.00 MiB, V (f16): 640.00 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 0.02 MiB, ( 151.78 / 10922.67)
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size = 2756.02 MiB, ( 2907.78 / 10922.67)
llama_new_context_with_model: graph splits (measure): 5
llama_new_context_with_model: Metal compute buffer size = 2756.01 MiB
llama_new_context_with_model: CPU compute buffer size = 2736.02 MiB
AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |

llama_print_timings: load time = 4014.04 ms
llama_print_timings: sample time = 21.10 ms / 208 runs ( 0.10 ms per token, 9858.29 tokens per second)
llama_print_timings: prompt eval time = 4013.94 ms / 35 tokens ( 114.68 ms per token, 8.72 tokens per second)
llama_print_timings: eval time = 17300.80 ms / 207 runs ( 83.58 ms per token, 11.96 tokens per second)
llama_print_timings: total time = 21616.44 ms / 242 tokens
{'question': 'Why is the sky blue?', 'text': "\n\n1. The Earth is surrounded by an atmosphere, which is made up of many gases including nitrogen and oxygen.\n2. When sunlight enters the Earth's atmosphere, it contains all the colors of the rainbow - red, orange, yellow, green, blue, indigo, and violet.\n3. As the sunlight passes through the atmosphere, some of the colors are scattered in different directions by the gas molecules. This is called scattering.\n4. The shorter wavelength colors (blue, indigo, and violet) are scattered more than the longer wavelength colors (red, orange, yellow, and green).\n5. When we look up at the sky, we see the blue color because our eyes are most sensitive to the blue light that has been scattered by the atmosphere.\n6. The other colors of the rainbow are also present in the sunlight, but they are not as easily seen because they are either absorbed by the atmosphere or scattered less than the blue light."}
ggml_metal_free: deallocating
```

The drop of performance is reproducible also with different prompts and in different applications outside of this example code. It seems to depend only on the version of llama-cpp-python. My workaround at the moment is setting

`llama-cpp-python = "<0.2.29"`

in my `Pipfile`. I also tried to run with the very latest version `0.2.32` and experienced the same result as with `0.2.29` . So it seems something has changed with version 0.2.29 which leads to this drop in performance running models on Mac GPUs with Metal.