nextcloud / nextcloud/context_chat_backend

check if llama.cpp is feasible to be used directly instead of llama-cpp-python

Open
#297 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement priority: normal
Dominant language
Python
Stars
27
Forks
24
Avg merge
3d 4h
Merged PRs (30d)
6

Description

for the docker builds, there are some limitations added by llama-cpp-python which prevent us from building llama.cpp with flexible support for SSE, AVX(2) and AVX512. They are not loaded on demand but hardcoded to support AVX2 or fail. If AVX512 were to be compiled (GGML_AVX512=ON), that build would fail on all systems that don't support it even if the system support AVX2 and the build has that compiled in.
the llama.cpp flags for flexible/dynamic build: GGML_BACKEND_DL=ON GGML_CPU_ALL_VARIANTS=ON
^ this may be addressed later on but not a priority.

for now the previous requirement of AVX2 has been kept intact.

Originally posted by @kyteinsky in https://github.com/nextcloud/context_chat_backend/issues/295#issuecomment-4498917560

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the Docker build configuration and the current llama-cpp-python dependency to compare its CPU support with llama.cpp's GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS flags. Done means determining whether a direct llama.cpp build can preserve AVX2 compatibility while enabling flexible SSE, AVX2, and AVX512 support, and documenting the feasibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
ai, build-system
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.