abetlen / abetlen/llama-cpp-python

Segmentation fault when converting embeddings into tensor

Aberta
#1,937 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
10.6k
Forks
1.4k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

trying to convert embedding into tensor leads to Segmentation fault:

---
> System Info
* Physical (or virtual) hardware you are using, e.g. for Linux:

```
> sysctl -a | grep machdep.cpu
machdep.cpu.cores_per_package: 10
machdep.cpu.core_count: 10
machdep.cpu.logical_per_package: 10
machdep.cpu.thread_count: 10
machdep.cpu.brand_string: Apple M2 Pro

```

* Operating System, e.g. for Linux:

`macos Sequia 15.3.1 (24D70)`

* SDK version, e.g. for Linux:

```
Python 3.13.2
GNU Make 3.81
Apple clang version 16.0.0 (clang-1600.0.26.6)
Target: arm64-apple-darwin24.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```

---

code
```python
import logging
import torch
from llama_cpp import Llama
from rich.console import Console

# Configure logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
console = Console(width=120)

embpath = "all-MiniLM-L6-v2-ggml-model-f16.gguf"
embedModel = Llama(model_path=embpath,embedding=True,verbose=True)

# test embedding model
query = ["Test sentence"]
try:
embeds = embedModel.embed(input=query)
print(embeds)
genAns_tensor = torch.tensor(embeds)

del embedModel
except Exception as e:
print("Embedding error:", e)
```
> code works for only creating embeddings ( i.e if i remove the tensor conversion part and just print the embedding)

logs
```text
llama_kv_cache_init: kv_size = 512, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 6, can_shift = 1
llama_kv_cache_init: layer 0: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 1: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 2: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 3: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 4: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: layer 5: n_embd_k_gqa = 384, n_embd_v_gqa = 384
llama_kv_cache_init: Metal KV buffer size = 4.50 MiB
llama_init_from_model: KV self size = 4.50 MiB, K (f16): 2.25 MiB, V (f16): 2.25 MiB
llama_init_from_model: CPU output buffer size = 0.00 MiB
llama_init_from_model: Metal compute buffer size = 17.00 MiB
llama_init_from_model: CPU compute buffer size = 3.50 MiB
llama_init_from_model: graph nodes = 221
llama_init_from_model: graph splits = 2
Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | MATMUL_INT8 = 1 | DOTPROD = 1 | MATMUL_INT8 = 1 | ACCELERATE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
Model metadata: {'tokenizer.ggml.cls_token_id': '101', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.seperator_token_id': '102', 'tokenizer.ggml.unknown_token_id': '100', 'tokenizer.ggml.token_type_count': '2', 'general.file_type': '1', 'tokenizer.ggml.eos_token_id': '102', 'bert.context_length': '512', 'bert.pooling_type': '1', 'tokenizer.ggml.bos_token_id': '101', 'bert.attention.head_count': '12', 'bert.feed_forward_length': '1536', 'tokenizer.ggml.mask_token_id': '103', 'tokenizer.ggml.model': 'bert', 'bert.attention.causal': 'false', 'general.name': 'all-MiniLM-L6-v2', 'bert.block_count': '6', 'bert.attention.layer_norm_epsilon': '0.000000', 'bert.embedding_length': '384', 'general.architecture': 'bert'}
Using fallback chat format: llama-2
Fatal Python error: Segmentation fault

Thread 0x0000000204908840 (most recent call first):
File "/Users/devashishraj/Desktop/localRAG/lrag/lib/python3.13/site-packages/llama_cpp/_internals.py", line 306 in decode
File "/Users/devashishraj/Desktop/localRAG/lrag/lib/python3.13/site-packages/llama_cpp/llama.py", line [1] 63839 segmentation fault PYTHONFAULTHANDLER=1 python3 -X dev embeddingTest.py
```

venv package list
```
pip list
Package Version
------------------------ -----------
aiohappyeyeballs 2.4.4
aiohttp 3.11.10
aiosignal 1.3.2
annotated-types 0.7.0
anyio 4.7.0
attrs 24.3.0
beautifulsoup4 4.12.3
certifi 2024.12.14
charset-normalizer 3.4.0
dataclasses-json 0.6.7
diskcache 5.6.3
faiss-cpu 1.9.0.post1
filelock 3.17.0
frozenlist 1.5.0
fsspec 2025.2.0
gpt4all 2.8.2
h11 0.14.0
httpcore 1.0.7
httpx 0.28.1
httpx-sse 0.4.0
huggingface-hub 0.28.1
idna 3.10
Jinja2 3.1.5
joblib 1.4.2
jsonpatch 1.33
jsonpointer 3.0.0
langchain 0.3.12
langchain-community 0.3.12
langchain-core 0.3.33
langchain-ollama 0.2.3
langchain-text-splitters 0.3.3
langsmith 0.2.3
llama_cpp_python 0.3.7
markdown-it-py 3.0.0
MarkupSafe 3.0.2
marshmallow 3.23.1
mdurl 0.1.2
mpmath 1.3.0
multidict 6.1.0
mypy-extensions 1.0.0
networkx 3.4.2
numpy 2.2.3
ollama 0.4.7
orjson 3.10.12
packaging 24.2
pillow 11.1.0
pip 25.0.1
propcache 0.2.1
pydantic 2.10.3
pydantic_core 2.27.1
pydantic-settings 2.7.0
Pygments 2.18.0
PyMuPDF 1.25.1
python-dotenv 1.0.1
PyYAML 6.0.2
regex 2024.11.6
requests 2.32.3
requests-toolbelt 1.0.0
rich 13.9.4
safetensors 0.5.2
scikit-learn 1.6.1
scipy 1.15.1
sentence-transformers 3.4.1
setuptools 75.8.0
sniffio 1.3.1
soupsieve 2.6
SQLAlchemy 2.0.36
sympy 1.13.1
tenacity 9.0.0
threadpoolctl 3.5.0
tiktoken 0.8.0
tokenizers 0.21.0
torch 2.6.0
tqdm 4.67.1
transformers 4.48.3
typing_extensions 4.12.2
typing-inspect 0.9.0
urllib3 2.2.3
yarl 1.18.3
```

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.