Memory allocation challenges with LlamaCppEmbeddings on M1 Mac

Hi, I am having problems with memory allocation warnings (that lead to crashes) when using LlamaCppEmbeddings on an M1 Mac. I am running llama-cpp-python v0.1.84 on a MacBook Pro with 16GB of RAM, which has 8 GPU cores.

I completed installation of llama-cpp-python using the instructions detailed below-- notably this command: CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python

https://github.com/abetlen/llama-cpp-python

I call LlamaCpp via langchain.llms and LlamaCppEmbeddings via langchain.embeddings.

When I set up the LlamaCpp function, all works as expected with the following parameters:

n_gpu_layers = 40 # Change this value based on your model and your GPU VRAM pool.
n_batch = 1024 # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.
# Callbacks support token-wise streaming
callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

llm = LlamaCpp(
model_path = "/Models/Llama2/llama-2-7b-chat.Q6_K.gguf",
n_gpu_layers=n_gpu_layers,
n_batch=n_batch,
n_ctx=2048,
callback_manager=callback_manager,
verbose=True, # Verbose is required to pass to the callback manager
)

Below is the last lines of the output I receive when running the above command, which is as I expected:

ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 10922.67 MB
ggml_metal_init: maxTransferRate = built-in GPU
llama_new_context_with_model: compute buffer total size = 169.47 MB
llama_new_context_with_model: max tensor size = 102.54 MB
ggml_metal_add_buffer: allocated 'data ' buffer, size = 5273.06 MB, ( 5273.50 / 10922.67)
ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1.48 MB, ( 5274.98 / 10922.67)
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 1026.00 MB, ( 6300.98 / 10922.67)
ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 168.02 MB, ( 6469.00 / 10922.67)
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |

I expected a similar output when running LlamaCppEmbeddings, but I got warnings about memory allocations. These are shown below:

ggml_metal_add_buffer: allocated 'data ' buffer, size = 5273.06 MB, (11742.06 / 10922.67), warning: current allocated size is greater than the recommended max working set size
ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1.48 MB, (11743.55 / 10922.67), warning: current allocated size is greater than the recommended max working set size
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 2050.00 MB, (13793.55 / 10922.67), warning: current allocated size is greater than the recommended max working set size
ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 168.02 MB, (13961.56 / 10922.67), warning: current allocated size is greater than the recommended max working set size
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |
>>>

If I try to run the command below to embed text, my IDE (RStudio) instantly crashes with a Fatal error -- which I presume to be related to memory.

docsearch = Chroma.from_documents(texts, embeddings)

I reset Python and tried running again -- but *not* running the LlamaCpp command above for llm.

This worked, as shown by the output below:

ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 10922.67 MB
ggml_metal_init: maxTransferRate = built-in GPU
llama_new_context_with_model: compute buffer total size = 169.47 MB
llama_new_context_with_model: max tensor size = 102.54 MB
ggml_metal_add_buffer: allocated 'data ' buffer, size = 5273.06 MB, ( 5273.50 / 10922.67)
ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1.48 MB, ( 5274.98 / 10922.67)
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 2050.00 MB, ( 7324.98 / 10922.67)
ggml_metal_add_buffer: allocated 'alloc ' buffer, size = 168.02 MB, ( 7493.00 / 10922.67)
AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |
>>>

However, RStudio (from which I'm running Python) instantly crashes when I try to run this command:

# Create vector database
docsearch = Chroma.from_documents(texts, embeddings)

My guess is that the crash is due to memory allocation issues. However, I can't find the settings that I should use with an M1 MacBook Pro. I have tried reducing n_gpu_layers from 40 to 1 and still got this error and outcome.

All works fine (but slowly) when using the CPU only--but I would ideally like to be able to use the GPUs on my MacBook Pro.

Can anyone advise?