nextcloud / nextcloud/context_chat_backend
check if llama.cpp is feasible to be used directly instead of llama-cpp-python
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 27
- Forks
- 24
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 6
Description
for the docker builds, there are some limitations added by llama-cpp-python which prevent us from building llama.cpp with flexible support for SSE, AVX(2) and AVX512. They are not loaded on demand but hardcoded to support AVX2 or fail. If AVX512 were to be compiled (GGML_AVX512=ON), that build would fail on all systems that don't support it even if the system support AVX2 and the build has that compiled in.
the llama.cpp flags for flexible/dynamic build: GGML_BACKEND_DL=ON GGML_CPU_ALL_VARIANTS=ON
^ this may be addressed later on but not a priority.
for now the previous requirement of AVX2 has been kept intact.
Originally posted by @kyteinsky in https://github.com/nextcloud/context_chat_backend/issues/295#issuecomment-4498917560
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the Docker build configuration and the current llama-cpp-python dependency to compare its CPU support with llama.cpp's GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS flags. Done means determining whether a direct llama.cpp build can preserve AVX2 compatibility while enabling flexible SSE, AVX2, and AVX512 support, and documenting the feasibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- ai, build-system
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100