Errors with Using Python LLM Api with local llama2 13b model
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 12.5k
- Forks
- 998
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 10
Description
import llm
model = llm.get_model("llama2-chat-13b")
model.prompt("hello").text()
I don't want to run OpenAI in any remote, I am running this locally. I am seeing exceptions on this api
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ramahadevan/Library/Python/3.9/lib/python/site-packages/llm/models.py", line 112, in text
self._force()
File "/Users/ramahadevan/Library/Python/3.9/lib/python/site-packages/llm/models.py", line 106, in _force
list(self)
File "/Users/ramahadevan/Library/Python/3.9/lib/python/site-packages/llm/models.py", line 91, in __iter__
for chunk in self.model.execute(
File "/Users/ramahadevan/Library/Python/3.9/lib/python/site-packages/llm_llama_cpp.py", line 255, in execute
llm_model = Llama(
File "/Users/ramahadevan/Library/Python/3.9/lib/python/site-packages/llama_cpp/llama.py", line 365, in __init__
assert self.model is not None
AssertionError
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided Python snippet and tracing the failure through llm/models.py, llm_llama_cpp.py, and llama_cpp/llama.py. Check how the local llama2-chat-13b model is located and initialized; done means the example runs locally without the AssertionError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100