"ggml_metal_free: deallocating" appended at the end of the responses
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.5k
- Forks
- 991
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 10
Description
I was setting up llm for the first time following https://simonwillison.net/2023/Aug/1/llama-2-mac/, but looks like llama.cpp has switched to GGUF model format, so I had to download the .gguf model instead.
llm llama-cpp download-model \
https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF/resolve/main/llama-2-7b-chat.Q8_0.gguf \
--alias llama2-chat --alias l2c --llama2-chat
But then it looks like the ggml_metal_free: deallocating string is appended to the responses, e.g.
❯ llm -m l2c 'Tell me a joke about a llama' --system 'You are Jerry Seinfeld'
(In my best Jerry Seinfeld voice) Oh, boy...llamas. You know, I've been thinking, have you ever noticed how llamas are like the ultimate pretenders? They're always standing there, looking all regal and noble, but deep down they're just a bunch of woolly impostors. (chuckles) I mean, have you seen their ears? They're like little floppy antennae, trying to listen in on all the juicy gossip. (giggles) And don't even get me started on their spittingggml_metal_free: deallocating
❯ llm -c 'Now be George'
Oh boy, oh boy! *excitedly* Llamas?! (clears throat) Well, you know what they say, "A llama in the hand is worth two in the bush!" *winks* But seriously, have you ever seen those fluffy creatures try to cross a busy street? They're like little traffic cones, trying to figure out which way the cars are gonna go! (chuckles) And don't even get me started on their love of soccer. I mean, they're always kicking the ball around, but they never scoreggml_metal_free: deallocating
Though looking at the logs, it seems the responses do not have this string, so I suspect that it is somehow leaked from llama.cpp to STDOUT.
Here are the versions and some related info:
❯ llm --version
llm, version 0.11
❯ llm plugins
[
{
"name": "llm-llama-cpp",
"hooks": [
"register_commands",
"register_models"
],
"version": "0.2b0"
}
]
❯ llm models
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
OpenAI Chat: gpt-4 (aliases: 4, gpt4)
OpenAI Chat: gpt-4-32k (aliases: 4-32k)
OpenAI Completion: gpt-3.5-turbo-instruct (aliases: 3.5-instruct, chatgpt-instruct)
LlamaModel: llama-2-7b-chat.Q8_0 (aliases: llama2-chat, l2c)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the output with the documented llm llama-cpp download-model and llm -m l2c commands using the reported llm 0.11 and llm-llama-cpp 0.2b0 versions. Inspect the llama-cpp plugin's response and output handling to determine why ggml_metal_free: deallocating reaches the terminal response; done means the text no longer appears in responses while model output still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100