Add configurable inference fallback for self-hosted agents
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
### Problem
I run Buzz agents primarily against a self-hosted vLLM endpoint. If that
server is unavailable, the agent cannot respond because buzz-agent supports
only one configured inference provider.
### Requested behavior
Allow an agent to use an ordered fallback:
1. Primary: self-hosted OpenAI-compatible endpoint
2. Secondary: self-hosted OR cloud hosted OpenAI-compatible provider (Venice.ai DIEM Credits for example as a backstop)
Failover should occur for connection failures, timeouts, rate limits, and
server errors, with automatic recovery to the primary endpoint.
### Use case
This would support local-first inference for privacy and cost while retaining
agent availability during maintenance or hardware outages.
An external gateway such as LiteLLM can provide this today, but native support
would simplify deployment and configuration.
Contributor guide
Research direction
Start at the buzz-agent entry point and trace how its single inference provider is configured and how requests are handled. Define the ordered fallback configuration, failure classes, and primary recovery behavior described in the issue, then add coverage for failover and recovery; done means local-first inference remains available through the secondary provider when needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100