Feature Request: Intel ARC GPU Support for Local LLM Backend

# Feature Request: Intel ARC GPU Support for Local LLM Backend

## Overview

**Issue Type**: Feature Request
**Priority**: High (Intel Arc GPUs are increasingly popular)
**Affected Hardware**: Intel Arc B570 / A750 / A770 / etc.
**Current Status**: No acceleration available (CPU-only, ~3 tokens/s)
**Expected Performance**: 25-35 tokens/s with Vulkan backend

---

## Problem Description

### Current Situation

When using `llama.cpp (Local)` in CoPaw's llm_routing, the Python package `llama-cpp-python` only supports:
- NVIDIA CUDA
- Apple Metal
- **Intel ARC (Vulkan/DPC++)** **Missing!**

This results in extremely slow performance on systems with Intel Arc GPUs:

```bash
# Current CPU-only performance
Prompt Processing: 1.7 t/s
Generation Speed: 3.1 t/s
```

### Expected Performance with Vulkan

Using pre-built `llama.cpp` with Vulkan backend:

```bash
# With Intel Arc B570 + Vulkan --n-gpu-layers 99
Prompt Processing: 15+ t/s
Generation Speed: 25-35 t/s
GPU Memory Used: ~7GB (arc b570 10G VRAM)
```

**Performance Gain**: **~8-10x speedup** when offloading to GPU!

---

## Proposed Solutions

### Option A: Custom HTTP Provider (Recommended)

Add support for custom HTTP endpoints in `llm_routing`:

```json
"llm_routing": {
"enabled": true,
"mode": "local_first",
"local": {
"provider_id": "custom_http", // New provider type
"base_url": "http://127.0.0.1:8081/v1",
"model": "qwen3.5-9b"
}
}
```

**Benefits:**
- Compatible with llama.cpp's `llama-server` (which supports Vulkan natively)
- Flexible: users can run any compatible server
- Low implementation effort: reuse existing OpenAI API client code

### Option B: Auto-Detect External llama.cpp Binary

Similar to how some tools detect external binaries (ffmpeg, git, etc.):

```json
"providers": {
"llama-cpp-external": {
"enabled": true,
"cli_path": "C:\path\to\llama-cli.exe",
"auto_detect": true,
"backend_type": "vulkan|cuda|metal|cpu"
}
}
```

---

## Validation Data

### Tested Configuration

**Hardware:**
- CPU: Intel i5-13400F
- GPU: Intel Arc B570 (10GB VRAM)
- RAM: 32GB DDR5
- OS: Windows 10/11

**Software:**
- llama.cpp: b8575 (pre-built win-vulkan)
- Model: Qwen3.5-9B-Q5_K_M.gguf (6.58GB)

**Results:**
```
Without GPU acceleration (CPU): Generation 3.1 t/s
With Vulkan + --n-gpu-layers 99: Generation 25-35 t/s (8-10x faster)
```

**Command used for testing:**
```bash
llama-cli.exe -m Qwen3.5-9B-Q5_K_M.gguf ^
--n-gpu-layers 99 ^
--no-mmap ^
-t 0 ^
-c 4096 ^
--temp 0.7 ^
--mirostat 2 ^
-cnv ^
--color on ^
--flash-attn on
```

---

## Thank You!

Thank you for building CoPaw! This project already saves me hours of work every day. Adding Intel ARC support would make it perfect for my setup and help thousands of others too.

Happy to help test any implementations or provide further technical details!

---

**Posted by**: User with Intel i5-13400F + Arc B570
**Date**: 2026-03-29
**CoPaw Version**: Latest (as of March 2026)