terraphim / terraphim/terraphim-ai

feat: Add GGUF/llama-cpp backend to terraphim LLM proxy layer

Open
#538 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement multi-agent
Dominant language
Rust
Stars
62
Forks
5
Avg merge
2h 27m
Merged PRs (30d)
1

Description

Problem

terraphim-ai has no local inference capability. All LLM calls go through remote APIs (OpenRouter via genai). For machines without GPU or API access, there is no fallback. GGUF models for MedGemma are available (unsloth/medgemma-1.5-4b-it-GGUF, 11.8K downloads on HuggingFace).

Proposed Change

Add a terraphim_llm_local crate (or feature in terraphim_multi_agent) that wraps llama-cpp-rs for local GGUF inference. Implement the same LLM client trait so agents can transparently use local or remote models.

Key requirements:

  • CPU-only inference support (many dev machines lack GPU)
  • Automatic GGUF model download via hf-hub crate
  • Quantization variant selection (Q4_K_M ~2.5GB for 4B models, Q8_0 for higher quality)
  • Same trait interface as the remote genai client for seamless swapping

Scope

  • New crate crates/terraphim_llm_local/ or feature gate in terraphim_multi_agent
  • Dependencies: llama-cpp-rs, hf-hub (both approved)

Context

This is UPLIFT-5 from the medgemma-competition multi-agent integration plan. Local GGUF inference is essential for development workflows where remote API calls are slow or unavailable. The MedGemma 1.5-4b-it GGUF model is the primary target for local inference.

Related upstream issues: #534, #535, #536

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the existing remote genai client trait with the proposed new crates/terraphim_llm_local/ or feature in terraphim_multi_agent, then review the approved llama-cpp-rs and hf-hub dependencies. Done means CPU-only GGUF inference, automatic model download, quantization selection, and transparent compatibility with the existing client trait.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.