Ensure all retryable provider errors are retried with backoff and UI feedback
- 主要言語
- Rust
- スター
- 54.2k
- フォーク
- 6.2k
- 平均マージ
- 3日 2時間
- マージ済み PR(30日)
- 262
説明
## Summary
When a local model provider (e.g. Ollama) is fully stopped, sending a message leaves the chat in an "in flight" state for **~2 minutes** before a network error is finally shown.
## Environment / Diagnostics
- **Platform/OS:** Darwin (macOS Tahoe 26.5.2)
- **Goose Version:** v1.45.0 (Desktop UI)
- **Provider Used:** Local — Ollama (`localhost:11434`), model `gemma4:26b`
- **Extensions Enabled:** [all default] - NA
- **Relevant config:** `OLLAMA_TIMEOUT: '600'` this timeout is never reached on attempting to connect to a read ollama server
## Problem Statement
Stopping the local provider mid-session does eventually produce a correct, clear error message. The problem is the **latency before that message appears**: the user waits ~2 minutes staring at a "working" indicator for a failure condition (nothing listening on the port) that is very likely permanent and returns quickly.
## Two failure modes tested
| Case | How it was simulated | Result |
|---|---|---|
| **A. Model Unavailable** (EOL'd / uninstalled) | Point the request at a model name Ollama doesn't have (Ollama still running) | ✅ Fast, clear `model not found` (404) error. **No bug.** |
| **B. Provider Unavailable** (server stopped) | Stop the Ollama server so `localhost:11434` refuses connections | ⚠️ ~2 min "in flight", then a network error. **The bug.** |
## Steps to Reproduce
1. Configure Goose to use a local Ollama model and start a chat. Ensure the model responds before continuing.
2. Fully stop the Ollama server (e.g. `pkill ollama` / quit the menubar app) so `localhost:11434` refuses connections.
3. Send a message in the existing session.
4. Observe the "working" indicator persists ~2 minutes before the `Could not connect to localhost:11434` error appears.
Confirmed that curling the stopped server directly was fast, so the delay originates entirely inside Goose's retry logic, not the network. `curl http://localhost:11434/api/tags` -0s-> `curl: (7) Failed to connect to localhost port 11434 after 0 ms: Couldn't connect to server`
## Observed Behavior
- 2:53pm; Chat shows the "Goose is working on it" indicator for ~2 minutes.
- 2:55pm: the following error appears
> Network error: Could not connect to localhost:11434 — check your network connection and try again.
>
> Please resend your message to try again.
## Root Cause
**Connection-refused is mapped to `NetworkError`.**
`crates/goose-provider-types/src/errors.rs` (~L91–118) — a reqwest `is_connect()` error becomes:
```rust
ProviderError::NetworkError("Could not connect to localhost:11434 — check your network connection and try again.")
```
**`NetworkError` is classified as retryable.**
`crates/goose-provider-types/src/retry.rs` (~L99–108):
```rust
pub fn should_retry(error: &ProviderError, config: &RetryConfig) -> bool {
match error {
ProviderError::RateLimitExceeded { .. }
| ProviderError::ServerError(_)
| ProviderError::NetworkError(_) => true, // ← connection-refused lands here
...
}
}
```
This lumps a **permanent** failure (connection refused — nothing listening) together with **transient** ones (server not ready yet, connection reset).
**3. Goose gives Ollama a deliberately generous retry budget for any retryable error.**
`crates/goose-provider-types/src/retry.rs` (L8–11) / applied in `crates/goose-providers/src/ollama.rs` (~L397–405):
```rust
// Ollama-specific: large models can take 30-120s to load into memory,
// during which Ollama returns 500 errors. Use more retries with gradual backoff...
const OLLAMA_MAX_RETRIES: usize = 10;
const OLLAMA_INITIAL_RETRY_INTERVAL_MS: u64 = 2000;
const OLLAMA_BACKOFF_MULTIPLIER: f64 = 1.5;
const OLLAMA_MAX_RETRY_INTERVAL_MS: u64 = 15_000;
```
### Timing math to check the observed behavior
Per-attempt backoff `initial * 1.5^(n-1)`, capped at 15s: ~100s. Looks like what I saw.
## Expected Behavior
A connection-refused error should exit quickly, not follow the existing backoff logic for ollama model loading. Also, the "Goose is working" indicator should stop as soon as the error is known (we probably get this for free if we surface the error sooner).
コントリビューションガイド
評価
この issue はまだ評価されていません。