feat: retry with backoff and opt-in model fallback

Motivation

LiteLLM's core value proposition is request-level retries and fallback routing; Pydantic AI ships FallbackModel(default, fallbacks=..., fallback_on=...) (https://ai.pydantic.dev/api/models/fallback/) so an agent can drop from a primary to a backup model/provider on failure; LangChain has .with_retry() and .with_fallbacks() runnable wrappers. Chatlas has neither today. A transient 429/500/connection error in the middle of an overnight batch_chat() job, or partway through a large parallel_chat() run, currently just raises and is captured as an Exception per-conversation (see _parallel_chat_impl in chatlas/_parallel.py, which has on_error: Literal["return", "continue", "stop"] and a fixed-rate RateLimiter, but no retry-with-backoff — a failed request is simply recorded via ErrorController.record_error() and never resubmitted).

Note that chatlas already relies on the official openai/anthropic Python SDKs (constructed in chatlas/_provider_openai.py's ChatOpenAI() and chatlas/_provider_anthropic.py's ChatAnthropic()), which default to max_retries=2 internally and are configurable today by passing kwargs_client={"max_retries": ...} through to the underlying OpenAI()/Anthropic() constructor — this is undocumented and doesn't help higher-level orchestration failures (e.g. after SDK retries are exhausted, or provider-agnostic paths like ChatGoogle/ChatBedrockAnthropic). ellmer hit exactly this gap in tidyverse/ellmer#722 ("chat_openai() errors instead of backing off") and closed it by adding ellmer_req_robustify() in R/httr2.R, which wraps every provider's HTTP request with httr2::req_retry(max_tries = getOption("ellmer_max_tries", 3), is_transient = ..., retry_on_failure = TRUE), with some providers (e.g. provider-claude.R) overriding is_transient for provider-specific transient-error detection. Because ellmer's providers talk HTTP directly (no vendor SDK), this pattern doesn't map 1:1 onto chatlas, which should layer on top of — not duplicate — the vendor SDKs' own retry logic.

Proposed approach

import chatlas as ctl

# (a) Opt-in retry w/ backoff, layered above SDK-level retries, for
# orchestration-level failures (e.g. exhausted SDK retries, or providers
# like ChatGoogle without built-in retry semantics)
chat = ctl.ChatAnthropic()
chats = await ctl.parallel_chat(
    chat,
    prompts,
    on_error="return",
    retry=ctl.RetryPolicy(max_tries=3, backoff="exponential", base_delay=1.0),
)

# (b) Opt-in ordered fallback across models/providers
chat = ctl.ChatOpenAI(model="gpt-5", fallback=[ctl.ChatAnthropic(), ctl.ChatOpenAI(model="gpt-5-mini")])
chat.chat("Summarize this report")  # falls through the list on transient failure

Candidate implementation points: extend ErrorController in chatlas/_parallel.py to resubmit (rather than just record) on transient errors before giving up, since that's where unattended runs already live; and/or add a retry=/fallback= kwarg to Chat.__init__ in chatlas/_chat.py so it also covers plain .chat()/.stream() calls. A shared transient-error classifier (429, 502/503/504, connection errors — explicitly not generic 4xx) could live near chatlas/_provider.py's Provider ABC so all providers classify consistently.

Caution: LiteLLM's default behavior of retrying on some 4xx client errors has caused documented retry-storm cost spikes for users; chatlas should default to retrying only clearly transient errors (429 with backoff honoring Retry-After, 5xx, connection resets) and never blanket-retry 4xx.

Alternatives / prior art

• ellmer: ellmer_req_robustify() (R/httr2.R), options(ellmer_max_tries=), closed via tidyverse/ellmer#722.
• Pydantic AI: FallbackModel (https://ai.pydantic.dev/api/models/fallback/).
• LangChain: Runnable.with_retry() / .with_fallbacks().
• LiteLLM: Router with fallbacks=/num_retries=.

Open questions

• Does this belong on Chat.__init__, as a per-call kwarg to .chat()/.stream(), or scoped only to parallel_chat/batch_chat where unattended runs happen? (Could start scoped to parallel_chat, per ellmer's parallel_chat_structured_robust discussion in tidyverse/ellmer#628, and expand later.)
• Should fallback trigger only on transient errors, or also on content-policy refusals / empty completions?
• Should retry state/attempts be surfaced anywhere (e.g. a warning, a field on the returned Chat), given the "operator wants a stop reason" feedback seen on the related ellmer tool-loop budget issue (tidyverse/ellmer#958)?

Drafted from a competitive review of llm / Pydantic AI / LangChain / LiteLLM (July 2026); filed via Claude Code on behalf of @cpsievert.

Start by reading chatlas/_parallel.py, especially _parallel_chat_impl and ErrorController, then review Chat.init in chatlas/_chat.py and the Provider ABC in chatlas/_provider.py. Resolve whether the first scope is parallel_chat, all chat calls, or both, and define transient-error and fallback behavior before implementation. Done should include an agreed API and observable retry and fallback outcomes without retrying generic 4xx errors.