abetlen / abetlen/llama-cpp-python

Include in Readme how to Pass Custom Arguments to `llama_cpp.server` in Docker

Aberta
#1,029 6 comentários 0 reações 0 responsáveis Ver no GitHub
documentation question
Linguagem predominante
Python
Estrelas
10.6k
Forks
1.4k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

### Title:
Issue with Passing Custom Arguments to `llama_cpp.server` in Docker

#### Issue Description:
Hello `abetlen`,

I've been trying to use your Docker image `ghcr.io/abetlen/llama-cpp-python:v0.2.24` for `llama_cpp.server`, and I encountered some difficulties when attempting to pass custom arguments (`--n_gpu_layers 81`, `--chat_format chatml`, `--use_mlock False`) to the server through Docker.

#### Steps to Reproduce:
1. Pull the Docker image: `docker pull ghcr.io/abetlen/llama-cpp-python:v0.2.24`
2. Run the container with custom arguments:

```bash
docker run --rm -it -p 8000:8000 \
-v /home/jaredquek/text-generation-webui/models:/models \
-e MODEL=/models/tulu-2-dpo-70b.Q5_K_M.gguf \
--entrypoint uvicorn \
ghcr.io/abetlen/llama-cpp-python:v0.2.24 \
--factory llama_cpp.server.app:create_app --host 0.0.0.0 --port 8000 --n_gpu_layers 81 --chat_format chatml --use_mlock False
```

This results in an error: `Error: No such option: --n_gpu_layers`.

#### Expected Behavior:
I expected to be able to pass these arguments to the `llama_cpp.server` application inside the Docker container.

#### Actual Behavior:
The `uvicorn` command does not recognize these arguments as it's designed for the ASGI server, not the `llama_cpp.server` application.

#### Potential Solutions:
- Modify the Dockerfile or application configuration to accept these arguments.
- Provide guidance in Readme on how to correctly pass additional arguments or configure the server with these settings.

I would appreciate any assistance or guidance you could provide on this issue.

Thank you for your time and for maintaining this project.

Best regards.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.