abetlen / abetlen/llama-cpp-python

Include in Readme how to Pass Custom Arguments to `llama_cpp.server` in Docker

Đang mở
#1,029 6 bình luận 0 reaction 0 người được giao Xem trên GitHub
documentation question
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Title:
Issue with Passing Custom Arguments to `llama_cpp.server` in Docker

#### Issue Description:
Hello `abetlen`,

I've been trying to use your Docker image `ghcr.io/abetlen/llama-cpp-python:v0.2.24` for `llama_cpp.server`, and I encountered some difficulties when attempting to pass custom arguments (`--n_gpu_layers 81`, `--chat_format chatml`, `--use_mlock False`) to the server through Docker.

#### Steps to Reproduce:
1. Pull the Docker image: `docker pull ghcr.io/abetlen/llama-cpp-python:v0.2.24`
2. Run the container with custom arguments:

```bash
docker run --rm -it -p 8000:8000 \
-v /home/jaredquek/text-generation-webui/models:/models \
-e MODEL=/models/tulu-2-dpo-70b.Q5_K_M.gguf \
--entrypoint uvicorn \
ghcr.io/abetlen/llama-cpp-python:v0.2.24 \
--factory llama_cpp.server.app:create_app --host 0.0.0.0 --port 8000 --n_gpu_layers 81 --chat_format chatml --use_mlock False
```

This results in an error: `Error: No such option: --n_gpu_layers`.

#### Expected Behavior:
I expected to be able to pass these arguments to the `llama_cpp.server` application inside the Docker container.

#### Actual Behavior:
The `uvicorn` command does not recognize these arguments as it's designed for the ASGI server, not the `llama_cpp.server` application.

#### Potential Solutions:
- Modify the Dockerfile or application configuration to accept these arguments.
- Provide guidance in Readme on how to correctly pass additional arguments or configure the server with these settings.

I would appreciate any assistance or guidance you could provide on this issue.

Thank you for your time and for maintaining this project.

Best regards.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.