feat: Hermes Agent automated inference lifecycle — discover, validate, register, serve

Summary

Add an automated inference lifecycle to Hermes Agent that discovers, validates, registers, and serves local LLM models — turning any GPU-equipped machine into a self-managing inference node that earns revenue via x402 payments.

The agent monitors community signal (x/LocalLLaMA, HuggingFace trending) for new model releases, downloads promising GGUF candidates, benchmarks them against ToolCall-15 (or configurable eval suite), and if they pass a quality threshold, registers them as a ServiceOffer via ERC-8004 and exposes them for monetized inference via obol sell http.

This builds on PR #288's M1 (sell inference) infrastructure and the llama-server systemd integration validated during local testing.

Motivation

Today the path from "new model dropped" to "serving inference for revenue" is entirely manual:

1. Human spots a model on X/Reddit/HuggingFace
2. Human downloads the GGUF
3. Human runs benchmarks
4. Human configures llama-server
5. Human registers as ServiceOffer
6. Human runs obol sell http

Each step has friction and requires expertise. An agent with access to the local GPU can automate the entire pipeline, continuously optimizing which model it serves based on quality, speed, and community signal.

Validated Building Blocks

The following components have been tested and proven during prototyping:

Component	Status	Details
x-cli for X/Twitter signal	✅ Working	Searches x/LocalLLaMA community, fetches benchmarks
HuggingFace model discovery	✅ Working	API for GGUF file sizes, quant availability
llama-server with ROCm/CUDA	✅ Working	Systemd service, auto-restart, partial GPU offload
ToolCall-15 benchmark	✅ Working	15 scenarios, deterministic scoring, OpenAI-compatible
llama-bench for tok/s	✅ Working	Prompt processing + generation speed measurement
TurboQuant KV cache compression	✅ Working	3.5x compression, near-zero quality loss
obol sell http (PR #288)	✅ Merged	x402 payment gate on inference endpoint
ERC-8004 discovery	✅ Merged	Worker registration and discovery

Baseline Results

Qwen3.5-27B Q6_K on AMD RX 6800 XT (16GB VRAM):

• ToolCall-15: 15/15 (30/30 pts) — perfect score
• Generation: ~21 tok/s (partial offload ngl=35)
• Prompt processing: ~423 tok/s
• Context: 32K with TurboQuant q4_0 KV cache

Architecture

┌──────────────────────────────────────────────────────────────────┐
│  Hermes Agent — Inference Lifecycle Manager                       │
│                                                                    │
│  ┌────────────┐   ┌────────────┐   ┌────────────┐   ┌─────────┐ │
│  │  DISCOVER   │──▶│  VALIDATE   │──▶│  REGISTER   │──▶│  SERVE  │ │
│  │             │   │             │   │             │   │         │ │
│  │ x/LocalLLaMA│   │ Download    │   │ ERC-8004    │   │ llama-  │ │
│  │ HuggingFace │   │ llama-bench │   │ ServiceOffer│   │ server  │ │
│  │ X trending  │   │ ToolCall-15 │   │ OASF skills │   │ + obol  │ │
│  │ RSS/Atom    │   │ Custom eval │   │ + domains   │   │ sell    │ │
│  └────────────┘   └────────────┘   └────────────┘   └─────────┘ │
│         │                │                │               │        │
│         ▼                ▼                ▼               ▼        │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │  Model Registry (local SQLite)                              │  │
│  │  - model_id, gguf_path, quant, size_gb, vram_required      │  │
│  │  - toolcall15_score, tok_s_gen, tok_s_prompt                │  │
│  │  - discovered_at, validated_at, registered_at, serving_since│  │
│  │  - source_url, community_signal (likes, bookmarks, mentions)│  │
│  │  - status: discovered | downloading | validating |          │  │
│  │           passed | failed | registered | serving | retired  │  │
│  └─────────────────────────────────────────────────────────────┘  │
│                                                                    │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │  Hardware Profile (auto-detected)                           │  │
│  │  - gpu_model, vram_gb, gpu_backend (CUDA/ROCm/Metal)       │  │
│  │  - ram_gb, cpu_cores, disk_free_gb                          │  │
│  │  - max_ngl (computed from vram vs model size)               │  │
│  │  - estimated_tok_s (from llama-bench history)               │  │
│  └─────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘

Phase 1: DISCOVER

The agent periodically scans for new model candidates using configurable signal sources.

Signal Sources (configurable)

inference_lifecycle:
  discover:
    sources:
      - type: x_community
        community_id: "1958212555958022177"  # x/LocalLLaMA
        scan_interval: 6h
        min_engagement:
          likes: 100
          bookmarks: 50
      - type: x_accounts
        accounts:
          - sudoingX
          - stevibe
          - 0xSero
          - TheAhmadOsman
        scan_interval: 3h
      - type: huggingface
        filters:
          library: gguf
          sort: trending
          min_downloads: 1000
        scan_interval: 12h
    filters:
      max_size_gb: 25          # must fit in VRAM (with partial offload headroom)
      required_keywords:
        - tool_calling OR agent OR function_calling
        - GGUF
      architecture_preference:
        - dense              # prefer dense over MoE for agent work
        - moe                # MoE acceptable if activation < 5B params

Discovery Logic

1. Scan each source on its interval
2. Extract model identifiers (HuggingFace repo + quant filename)
3. Check against local model registry — skip if already known
4. Score signal strength: signal = likes * 1 + bookmarks * 2 + quote_tweets * 3
5. If signal > threshold AND model fits hardware profile → add to download queue
6. Agent can also be prompted manually: "check out this model: [URL]"

Phase 2: VALIDATE

Download and benchmark each candidate against configurable eval suites.

Validation Pipeline

1. Pre-flight check
   - Verify disk space (model_size * 1.5 for safety)
   - Verify VRAM budget (model + KV cache + overhead)
   - Compute optimal ngl (GPU layers) from VRAM budget

2. Download
   - wget/huggingface-cli to ~/Models/gguf/
   - Verify file integrity (expected size)
   
3. Speed benchmark (llama-bench)
   - pp512 (prompt processing speed)
   - tg128 (token generation speed)
   - Record in model registry

4. Quality benchmark (ToolCall-15)
   - Start temporary llama-server on ephemeral port
   - Run ToolCall-15 suite (15 scenarios, temp=0)
   - Record per-scenario pass/partial/fail + total score
   - Kill temporary server

5. Verdict
   - PASS: score >= threshold (configurable, default 14/15)
   - FAIL: score < threshold → mark as failed, optionally delete GGUF
   - If PASS and score > current serving model → candidate for promotion

Validation Config

inference_lifecycle:
  validate:
    eval_suite: toolcall-15          # or custom eval path
    pass_threshold: 14               # out of 15
    promote_threshold: 15            # must beat current to auto-promote
    min_tok_s: 10                    # minimum generation speed
    max_concurrent_validations: 1    # one at a time (GPU contention)
    cleanup_failed: true             # delete GGUFs that fail
    llama_server_flags:
      jinja: true
      ctk: q4_0                     # TurboQuant KV cache
      ctv: q4_0

Phase 3: REGISTER

Models that pass validation are registered as available inference endpoints.

Registration Flow

1. Update local model registry: status = registered
2. If obol-stack is configured:
   • Create/update ServiceOffer CRD with OASF metadata
   • Register on ERC-8004 with model capabilities
   • OASF skill: natural_language_processing/natural_language_generation/text_completion
   • OASF domain: configured per use case
3. Update Hermes config.yaml with new model as available provider
4. Notify operator (Telegram/Discord) with benchmark results

Registration Config

inference_lifecycle:
  register:
    auto_register_erc8004: true
    oasf:
      skill: natural_language_processing/natural_language_generation/text_completion
      domain: technology/data_science
    notify:
      channel: telegram
      on_new_model: true
      on_promotion: true
      include_benchmark_results: true

Phase 4: SERVE

Start or hot-swap the model for live inference, optionally monetized via x402.

Serving Modes

inference_lifecycle:
  serve:
    mode: single                     # single | multi | champion-challenger
    # single: one model at a time, hot-swap on promotion
    # multi: multiple models on different ports
    # champion-challenger: serve current best, test new on shadow traffic
    
    server:
      binary: llama-server           # from llama.cpp build
      host: 0.0.0.0
      port: 8080
      threads: 16
      context: 32768
      systemd: true                  # manage as systemd service
      
    monetize:
      enabled: true
      price_per_request: "0.001"     # USDC via x402
      obol_sell: true                # use obol sell http

Hot-Swap Logic

When a new model scores higher than the current serving model:

1. Notify operator: "Model X scored 15/15 vs current Y at 14/15. Promote?"
2. If auto_promote enabled OR operator confirms:
   • Stop current llama-server
   • Update systemd service with new model path + optimal ngl
   • Start new llama-server
   • Run quick health check (1 test request)
   • Update ServiceOffer if ERC-8004 registered
   • Log promotion in model registry
3. If new model fails health check → rollback to previous model

API Interface (obol-stack internal)

The inference lifecycle is exposed exclusively as an internal API within obol-stack, consumed by the obol-agent via skills. No standalone CLI — all operations are agent-driven through skill invocations.

obol-stack API Endpoints

# Discovery
POST /api/v1/inference/discover              # trigger discovery scan
GET  /api/v1/inference/discover/candidates   # list discovered candidates

# Validation  
POST /api/v1/inference/validate              # validate a specific model
     body: { url: "hf://unsloth/Qwen3.5-27B-GGUF/Q6_K", eval_suite: "toolcall-15" }
GET  /api/v1/inference/validate/:job_id      # poll validation status/results

# Registry
GET  /api/v1/inference/models                # list all models + scores + status
PUT  /api/v1/inference/models/:id/promote    # promote model to serving
PUT  /api/v1/inference/models/:id/retire     # retire model

# Serving
GET  /api/v1/inference/status                # current serving model + health + stats
POST /api/v1/inference/serve                 # hot-swap to specified model
     body: { model_id: "qwen35-27b-q6k", rollback_on_failure: true }

# Hardware
GET  /api/v1/inference/hardware              # auto-detected hardware profile

obol-agent Skill Integration

The agent interacts with the lifecycle via a dedicated inference-lifecycle skill:

# Skill: inference-lifecycle
# Registered in obol-stack ServiceOffer CRD
# Agent can be instructed conversationally or via cron:

# Examples of agent-driven operations:
# "Scan x/LocalLLaMA for new models worth testing"
#   → skill calls POST /api/v1/inference/discover
#
# "Validate this model: hf://unsloth/Qwen3.5-27B-GGUF/Q6_K"  
#   → skill calls POST /api/v1/inference/validate
#
# "What's our current model performance?"
#   → skill calls GET /api/v1/inference/status
#
# "Promote the Nemotron model if it scored higher"
#   → skill calls PUT /api/v1/inference/models/:id/promote

Configuration (via obol-stack values.yaml)

inferenceLifecycle:
  enabled: true
  hardwareProfile: auto              # auto-detect GPU/RAM/disk
  modelsDir: /models/gguf
  registryDb: /data/inference-registry.db
  discover:
    # ... sources config
  validate:
    # ... eval config  
  register:
    # ... registration config
  serve:
    # ... serving config

Dependency on PR #288

This feature depends on PR #288 for:

• obol sell http — monetized inference endpoint
• ServiceOffer CRD — Kubernetes-native model registration
• x402 payment parsing — parse_402_pricing for per-request billing
• ERC-8004 discovery — workers find our inference endpoint
• --register-skills / --register-domains — OASF metadata in ServiceOffer

New additions beyond PR #288:

• Automated model discovery from social/community signal
• Automated benchmark validation pipeline
• Model registry with scoring history
• Hot-swap serving with rollback
• Hardware-aware model selection (VRAM budget, optimal ngl)

Test Plan

Unit Tests

• Discovery: parse x-cli tweet output → extract model candidates
• Discovery: HuggingFace API → filter by GGUF, size, trending
• Discovery: signal scoring (likes, bookmarks, quotes) → ranked list
• Validation: hardware profile → compute max model size + optimal ngl
• Validation: ToolCall-15 result parsing → pass/fail/partial
• Registry: model state machine (discovered → downloading → validating → passed/failed → registered → serving → retired)
• Serving: systemd service generation from model config
• Serving: hot-swap with rollback on health check failure

Integration Tests

• End-to-end: discover → download → benchmark → register → serve
• Hot-swap: promote new model, verify old model stops, new model serves
• Rollback: promote bad model, health check fails, previous model restored
• x402: serve model via obol sell http, verify payment flow
• ERC-8004: register model, discover from another node

Manual Validation (done during prototyping)

• x-cli sources x/LocalLLaMA community signal
• llama-bench measures tok/s on AMD ROCm
• ToolCall-15 runs against local llama-server (Qwen3.5-27B Q6_K = 15/15)
• llama-server as systemd service (auto-restart, survives reboot)
• TurboQuant KV cache compression works on ROCm

Labels

component:inference component:hermes-agent priority:high size:L

Begin with obol-stack values.yaml and the listed /api/v1/inference endpoints, then inspect the existing obol sell http and llama-server integrations referenced in the issue. Map the discovery, validation, registry, hardware, and serving flows before choosing an implementation boundary. Done means the lifecycle is exposed through the internal API, persists model state, and supports the configured agent-driven operations.