block / block/buzz

Add configurable inference fallback for self-hosted agents

Open
#3,203 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

### Problem

I run Buzz agents primarily against a self-hosted vLLM endpoint. If that
server is unavailable, the agent cannot respond because buzz-agent supports
only one configured inference provider.

### Requested behavior

Allow an agent to use an ordered fallback:

1. Primary: self-hosted OpenAI-compatible endpoint
2. Secondary: self-hosted OR cloud hosted OpenAI-compatible provider (Venice.ai DIEM Credits for example as a backstop)

Failover should occur for connection failures, timeouts, rate limits, and
server errors, with automatic recovery to the primary endpoint.

### Use case

This would support local-first inference for privacy and cost while retaining
agent availability during maintenance or hardware outages.

An external gateway such as LiteLLM can provide this today, but native support
would simplify deployment and configuration.

Contributor guide

Open the contributing guide

Research direction

Start at the buzz-agent entry point and trace how its single inference provider is configured and how requests are handled. Define the ordered fallback configuration, failure classes, and primary recovery behavior described in the issue, then add coverage for failover and recovery; done means local-first inference remains available through the secondary provider when needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.