googleapis / googleapis/python-genai

first-class model/region fallbacks

Open
#1,288 1 comment 2 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
api: gemini-api priority: p3 type: feature request
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

Thanks for stopping by to let us know something could be better!

**PLEASE READ**: If you have a support contract with Google, please create an issue in the [support console](https://cloud.google.com/support/) instead of filing on GitHub. This will ensure a timely response.

**Is your feature request related to a problem? Please describe.**
In production we occasionally hit transient failures with Gemini and Model Garden (Vertex) models—e.g., `429 RESOURCE_EXHAUSTED`, `503 UNAVAILABLE`, or regional hiccups. While the SDK lets us pick a `location` (including Vertex regional and `global`) and exposes `APIError`, there’s no first-class mechanism to automatically fall back to:
- another **model version** (e.g., `gemini-2.5-pro` → `gemini-2.0-flash-001`), and/or
- another **location** (e.g., `us-central1` → `global`), or to a **partner model** in Model Garden,
based on well-defined error conditions. We currently implement this ourselves with orchestration libs or custom wrappers, which leads to inconsistent behavior and boilerplate.

**Describe the solution you'd like**
Add an **opt-in fallback policy** to the SDK that can be set globally on `genai.Client(...)` and/or per-call:

```python
from google import genai
from google.genai import types

policy = types.FallbackPolicy(
order=[
types.FallbackTarget(model="gemini-2.5-pro", location="us-central1"),
# Prefer global only if allowed by policy (no residency requirement):
types.FallbackTarget(model="gemini-2.5-pro", location="global"),
# Optional: partner model via Model Garden / publisher switch:
types.FallbackTarget(publisher="anthropic", model="claude-3.7-sonnet", location="europe-west4"),
# Final downgrade:
types.FallbackTarget(model="gemini-2.0-flash-001", location="us-east4"),
],
trigger=types.FallbackTrigger(
statuses=[429, 500, 503],
canonical_codes=["RESOURCE_EXHAUSTED", "UNAVAILABLE"],
timeout_ms=30_000,
# Do NOT fallback on client mistakes or safety blocks:
exclude_codes=["INVALID_ARGUMENT", "FAILED_PRECONDITION", "SAFETY"]
),
backoff=types.Backoff(initial=0.5, multiplier=2.0, max=8.0, jitter=True),
max_attempts=4,
propagate_last_error=True,
emit_telemetry=True, # e.g., header note + structured logs: which target succeeded
)

client = genai.Client(
vertexai=True, project="my-proj", location="us-central1",
fallback=policy,
)

resp = client.models.generate_content(
model="gemini-2.5-pro",
contents="Draft a release note about feature X.",
# Per-call override/disable is also useful:
# fallback=policy.without_global()
)
print(resp.text)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.