metabase / metabase/metabase

Metabot agent loop discards all progress when hitting max-iterations — user gets an error despite full token spend

Open
#82,181 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

.Team/Metabot Priority:P3
Dominant language
Clojure
Stars
49.3k
Forks
6.8k
Avg merge
1d 13h
Merged PRs (30d)
653

Description

## Describe the bug

When Metabot's agent loop reaches its iteration limit (hardcoded, currently 10) before the model emits a final text response, the loop exits with `:reason :max-iterations` and the user is shown a generic error. No answer is rendered — not even a partial one.

The problem is that by this point the loop has already made 10 LLM calls (all billed) and executed a series of successful tool calls (`read_resource`, `load_skill`, …) whose results are sitting in the message history. All of that context is silently discarded. From the user's perspective: tokens were consumed, nothing came back.

This is easy to hit with a self-hosted LLM provider (configured per #72458) on instances whose metadata is still sparse (e.g. a freshly connected database mid-sync), because the model needs more exploration hops to orient itself — but any question requiring more than ~9 tool calls will reproduce it regardless of provider.

## To Reproduce

1. Configure a self-hosted LLM provider (`MB_LLM_METABOT_PROVIDER`, `MB_LLM_ANTHROPIC_API_BASE_URL`, `MB_LLM_ANTHROPIC_API_KEY`).
2. Connect a database whose metadata sync is incomplete, or ask about entities the model has to discover.
3. Ask Metabot a question that requires several `read_resource` / `load_skill` hops.
4. The model spends every iteration on tool calls (each iteration's parts are `[:start :tool-input :usage :tool-output]` — never `:text`), the loop exits at iteration 10, and the UI shows an error with no answer.

## Logs

Sanitized excerpt (v0.63.15.4, provider `anthropic` via an Anthropic-compatible proxy; model alias redacted):

```
DEBUG agent.core :: Iteration {:n 9, :parts-count 4}
DEBUG agent.core :: Got parts {:count 4, :types [:start :tool-input :usage :tool-output]}
INFO metabot.self :: Calling LLM {:provider anthropic, :model claude-, :parts 19, :tools 13, :tool-choice nil, :ai-proxy? false}
INFO self.claude :: :metabot.claude/request (1277.221207ms) {:model "claude-", :msg-count 19, :tool-count 13}
DEBUG self.core :: Executing tool {:tool-name read_resource}
INFO self.core :: :metabot.agent/run-tool (6.034358ms) {:tool-name "read_resource", :tool-call-id "toolu_01J4D2CJXJJDvTVjBhe39XVu"}
DEBUG agent.core :: Iteration {:n 10, :parts-count 4}
DEBUG agent.core :: Got parts {:count 4, :types [:start :tool-input :usage :tool-output]}
INFO agent.core :: Agent loop complete {:iterations 10, :reason :max-iterations}
INFO agent.core :: :metabot.agent/run-agent-loop (26939.124726ms) {:profile-id :internal, :msg-count 1}
DEBUG middleware.log :: POST /api/metabot/agent-streaming 202 [ASYNC: completed] 26961ms (87 DB calls)
```

Note that every iteration produced a tool call and no `:text` parts, so the stream delivered to the frontend contains nothing renderable.

## Expected behavior

Two proposals (happy to split into separate issues if preferred):

1. **Graceful finalization on budget exhaustion.** When the loop is about to exit with `:reason :max-iterations`, make one final LLM call with tools disabled (`tool_choice: none` or equivalent) and a short instruction to answer with what has been gathered so far. The message history already contains every tool result, so this reliably produces a useful — if partial — answer for the cost of a single additional call, instead of discarding the 10 calls already paid for. Ideally the response is flagged so the UI can indicate it may be incomplete ("I ran out of exploration budget; here's my best answer so far").

2. **Make the iteration limit configurable.** The limit appears to be hardcoded in `metabase.metabot.agent.core`. A `defsetting` (env-settable, e.g. `MB_METABOT_AGENT_MAX_ITERATIONS`) would let self-hosted operators trade latency/cost for completion rate. The default of 10 is easily exhausted by models that prefer one tool call per turn.

Proposal 1 is the important one: the current behavior has the worst possible cost profile — maximum token spend, zero user value.

## Info

- Metabase version: v0.63.15.4 (self-hosted, Docker)
- LLM: Anthropic Claude via a self-hosted OpenAI/Anthropic-compatible proxy (LiteLLM), configured through the env vars from #72458
- Reproduces consistently on questions requiring exploration; unrelated to transport (verified: streaming, tool-call parsing, and tool execution all succeed — the loop simply runs out of iterations)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in metabase.metabot.agent.core at the agent loop and its max-iterations exit, then trace how the message history and stream parts reach the UI. Reproduce the no-text tool-call path with the provided logs and inspect the existing agent-loop tests or entry points. Done means max-iteration exhaustion no longer leaves the user with only a generic error, with the chosen behavior covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.