abetlen / abetlen/llama-cpp-python

Multi-GPU error

Open
#887 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

There are two AMDW6800 graphics cards on the current machine.
python3 -m llama_cpp.server --model models/codellama-13b-instruct.Q5_K_M.gguf --n_gpu_layers 45

ggml_cuda_set_main_device: using device 0 (AMD Radeon PRO W6800) as main device
llm_load_tensors: mem required = 107.59 MB (+ 1600.00 MB per state)
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloading v cache to GPU
llm_load_tensors: offloading k cache to GPU
llm_load_tensors: offloaded 43/43 layers to GPU
llm_load_tensors: VRAM used: 10295 MB
...................................................................................................
llama_new_context_with_model: kv self size = 1600.00 MB
llama_new_context_with_model: compute buffer total size = 191.47 MB
llama_new_context_with_model: VRAM scratch buffer: 190.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
INFO: Started server process [20048]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)

Successfully started
But the front-end randomly types a few words to send, and the back-end directly Segmentation fault (core dumped)
Normal operation with a single GPU

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.