Local LLM Infrastructure for PR Code Review Pipeline

Local LLM Infrastructure for PR Code Review Pipeline

Summary

Research findings and architectural recommendations for replacing/supplementing Claude Opus API usage in the automated PR deep-review pipeline with local LLM inference, eliminating daily rate limit outages and reducing API costs.

---

Problem Statement

The current review pipeline runs 4 parallel Opus 4.6 agents (each consuming ~75k tokens) per review, firing 6–10 reviews per day. This produces:

• Daily rate limit outages hitting 2–3 hours before the 5-hour window resets — sometimes multiple times per day
• ~444k tokens consumed per review burst against a Max 5x window of ~88k tokens — a single review exceeds the window budget
• ~$898/month estimated API cost at Opus 4.6 pricing ($5/$25 per MTok) if moved to pure API billing
• Code leaves the network on every triage agent call, including for work that doesn't require frontier reasoning

---

Findings

Hardware Recommendation: Apple M4 Max 128GB (~$4,000)

Why Apple Silicon beats x86 for this use case:

Architecture	Memory Bandwidth	70B Q4 Speed	Unified Memory	Cost
M4 Max 128GB	400 GB/s	~10 tok/sec	✅ 128GB all accessible	~$4,000
M3 Ultra 192GB	800 GB/s	~20 tok/sec	✅ 192GB all accessible	~$7,500
A100 80GB server	2.0 TB/s (VRAM only)	~15 tok/sec	❌ Split CPU/GPU pools	~$15,000–20,000
Dual EPYC CPU server	~920 GB/s	~1–3 tok/sec	❌ CPU matrix ops, no tensor	~$10,000–15,000

KV cache at 75k tokens (per agent, 70B GQA model):

~24.6 GB per agent instance
4 agents sequential: 40GB weights + 25GB KV = ~65GB → fits M4 Max 128GB comfortably
4 agents parallel: 40GB weights + 4×25GB KV = ~140GB → requires M3 Ultra 192GB

Sequential agent execution on M4 Max 128GB is the correct fit — agents are already async background jobs.

Wall clock time at 75k tokens, 70B Q4, ~3,000 token output:

Prefill:   75,000 / 2,000 tok/sec = ~38 sec
Generate:   3,000 / 10 tok/sec   = ~300 sec
Total per agent:                  ~5.5 min
4 agents sequential:              ~22 min

Reviews run as background jobs; 22 min wall clock is acceptable.

TCO over 24 months:

M4 Max 128GB:          $4,000 (no ongoing cost)
A100 colo server:      ~$16,000 hardware + ~$6,000 colo = ~$22,000
Payback vs API hybrid: ~5 months

---

Hybrid Architecture: Local Triage + Opus Synthesis

PR Created (Azure DevOps webhook)
         │
         ▼
  Orchestrator (Azure Function / MCP server)
         │
    ┌────┴──────────────┐
    │                   │
    ▼                   ▼
Agent 1   Agent 2   Agent 3  ──► Local Ollama on M4 Max
(style)   (security) (tests)      70B model (Qwen2.5-Coder 72B)
                                  $0 API cost, on-prem
                                  Results → distilled findings
    └────────┬──────────┘
             │
             ▼
         Agent 4 ──► Opus 4.6 synthesis
                      Receives distilled findings
                      Full cross-agent reasoning
                      ~$0.60/review fully optimized
                             │
                             ▼
                   Post review comment to PR

Recommended local model: Qwen2.5-Coder 72B Q4_K_M

• Best-in-class for code tasks at 70B scale
• 75k context well within reliable reasoning window
• ~40GB weights, fits M4 Max with ~63GB remaining for KV cache + OS

---

API Cost Impact (April 2026 Pricing)

Current models: Opus 4.6 at $5/$25 per MTok input/output. Extended thinking billed as output tokens.

Configuration	Per Review	Monthly (176 reviews)
4× Opus 4.6 + thinking (current)	~$5.10	~$898
4× Opus 4.6, no thinking	~$2.10	~$370
3× local + 1× Opus synthesis	~$1.275	~$224
3× local + 1× Opus, no thinking	~$0.525	~$92
Fully optimized (+ caching + batch)	~$0.60	~$106

API optimization levers available (stackable):

1. Prompt caching — system prompts are identical per agent; cache hits cost 10% of input price → ~24% input cost reduction
2. Batch API — 50% discount for async processing; PR reviews don't need real-time streaming
3. Model tiering — Haiku 4.5 ($1/$5) for pure syntax/style agents where Opus reasoning is wasted

M4 Max payback calculation:

Monthly saving (API-only vs hybrid):   ~$792/month
Annual saving:                         ~$9,504/year
Hardware cost:                         ~$4,000
Payback period:                        ~5 months

---

Rate Limit Impact

Current state on Max 5x ($100/month):

5-hour window budget:   ~88k tokens
Tokens per review:      ~444k (4 agents × 75k input + thinking)
Reviews per window:     0.2  ← single review exceeds the window

With hybrid (3× local + 1× Opus):

Opus tokens per review: ~85k (synthesis agent only)
Reviews per window:     ~1.0
Daily capacity before throttle: ~4× current

Moving to pure API billing for the PR pipeline eliminates subscription rate limits entirely — API has its own (much higher) tier-based limits that scale with spend.

---

Scalability

Horizontal scaling is near-perfectly linear:

• Each M4 Max is an independent Ollama inference endpoint
• No inter-unit communication required (unlike tensor parallelism)
• Load balancer (nginx round-robin) distributes requests across units

Units	Concurrent Reviews	Monthly Hardware Cost
1× M4 Max	1	$4,000 (one-time)
2× M4 Max	2	$8,000 (one-time)
3× M4 Max	3	$12,000 (one-time)

For a 4-developer team running 8 reviews/day each asynchronously, 2 units comfortably covers real-world concurrency. 3 units handles the full 10-person team scenario.

Phase 2 path: Dedicated Mac Studio (headless, colo or on-prem) as the inference server — removes workstation dependency, enables always-on availability for the team.

Phase 3 path: Aligns with existing RAG/MCP architecture — Azure Container Apps orchestrator routes to local Mac Studios (sensitive code) or Azure-hosted Ollama (overflow/less sensitive), transparent to agents via MCP connection string.

---

Validation Steps Before Buying Hardware

The following tests should gate the hardware purchase decision:

• Disable extended thinking on 3 triage agents (keep Opus) — measure quality delta. Cost: $0.
• Swap triage agents to Sonnet 4.6 (no thinking) — validate synthesis quality holds across 20+ reviews.
• Profile agent specialization — determine which of the 4 agents requires frontier reasoning vs. pattern matching.
• Run Qwen2.5-Coder 72B via Ollama locally (can test on any machine with sufficient RAM) — spot-check output quality on representative PRs before committing to hardware.

If Sonnet triage validates → M4 Max purchase is justified.
If Sonnet triage doesn't validate → upgrade to Max 20x ($200/month), drop thinking on triage agents (still 4× headroom improvement over current state for $100/month more).

---

Labels

infrastructure ai code-review cost-optimization local-llm

No repository files, tests, or implementation entry points are named. Start with the listed validation steps: measure current-agent alternatives and run Qwen2.5-Coder 72B through Ollama on representative pull requests. Done means establishing review quality, cost, rate-limit, and hardware findings before committing to an architecture or purchase.