mesa / mesa/mesa-llm

Issues with Gemini quickstart flow (retry behavior, model errors, and tool usage

Open
#266 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
73
Forks
89
Avg merge
14d 58m
Merged PRs (30d)
2

Description

Hi, I found a few issues while testing the quickstart flow with Gemini that seem worth addressing for first-run usability.

Tagging @jackiekazil and @colinfrisch in case this is helpful context.

---

## **Observed issues**

### Unbounded retry behavior in ModuleLLM
`ModuleLLM.generate()` / `agenerate()` retry transient LiteLLM errors such as `APIConnectionError`, `Timeout`, and `RateLimitError` using exponential backoff, but without a clear stop condition.

In practice, this can make the script appear stuck while LiteLLM repeatedly prints its help/debug banner.

---

### Outdated Gemini model names causing 404 errors
Older Gemini model names such as:
- `gemini-pro`
- `gemini-1.5-pro`

now return 404 errors from the Google API.

This currently surfaces as a raw LiteLLM traceback, which makes it harder for users to understand that the issue is an outdated model name rather than a runtime failure.

---

### Missing actionable error message for NotFoundError
It may be helpful to rewrite Gemini `NotFoundError` into a more actionable message, for example:

> Model 'gemini/gemini-1.5-pro' was not found. This Gemini model may no longer be available via this API. Try 'gemini/gemini-2.0-flash' instead.

---

### Tool usage in quickstart (`tool_choice="required"`)
It may also be worth reviewing whether `tool_choice="required"` in the CoT executor step is ideal for minimal examples.

In practice, it can make simple quickstart usage more dependent on tool-calling behavior than users might expect, and may introduce unnecessary failures for providers that do not fully support tool calling.

---

## **Possible improvements**

- Add a bounded retry policy in `ModuleLLM` so transient failures eventually surface clearly (instead of potentially looping indefinitely).
- Rewrite Gemini `NotFoundError` responses into a more actionable error message with a suggested replacement model.
- Update quickstart/default examples to use a currently available Gemini model (e.g. `gemini-2.0-flash`, depending on API availability).
- Optionally revisit whether `tool_choice="required"` is appropriate for minimal quickstart examples.

---

## **Example of clearer error messaging**

Something like:

> Model 'gemini/gemini-1.5-pro' was not found. This Gemini model may no longer be available via this API. Try 'gemini/gemini-2.0-flash' instead.

---

If this direction sounds reasonable, I’d be happy to work on a fix and open a PR.

Contributor guide

Open the contributing guide

Research direction

Start with ModuleLLM.generate() and agenerate() to trace the current retry behavior, then inspect the quickstart/default examples and the CoT executor's tool_choice="required" setting. Compare Gemini model handling for NotFoundError and define completion around bounded retries, actionable errors, current model examples, and an appropriate tool-use default.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.