simonw / simonw/llm

How to install models offline?

Open
#152 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted
Dominant language
Python
Stars
12.5k
Forks
998
Avg merge
3d 13h
Merged PRs (30d)
10

Description

I would like to use llm completely offline. I can install the llm python package offline with:

pip download llm

and then using this on the offline computer

pip install --no-index --find-links "$(convert-path($pwd))" .\llm-0.6.1-py3-none-any.whl

However, I get stuck once I want to install the offline models. To install them, I have to use:

llm install llm-gpt4all

then, when I actually specify the model I want to use (such as with llm -m ggml-vicuna-7b-1 "Five cut names for a pet penguin"), the .bin file gets downloaded here:

%userprofile%\.cache\gpt4all

I assume I can just copy the .bin model files in the .cache\gpt4all folder to the offline computer (but maybe I'm wrong, I'm not sure). However, I don't know how to replicate what the llm install llm-gpt4all line is doing completely offline. I used a drive monitoring program and it looks like it adds a bunch of files in the Python folder in the Program Files directory. For example:

image

It's not as easy to distinguish which of those are important compared to the single .bin file in the .cache folder.

Do you have any advice on how I can install these offline?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names the llm install llm-gpt4all command and the %userprofile%\.cache\gpt4all model directory, but no source files or tests. Start by tracing how the command installs the plugin and model, then document a complete offline procedure and verify it on the stated Windows setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, cli
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.