googlesamples / googlesamples/mlkit

Document AICore's per-app inference quota – sustained generation returns GenAiException BUSY (error code 9) after ~40 requests

Open
#1,070 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Kotlin
Stars
4.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

### Description

Sustained back-to-back inference through the Prompt API hits a rolling per-app quota that is not documented anywhere. After roughly 36–41 consecutive generations (~1.5 s each, so about a minute of sustained load), every subsequent request fails fast (~15–25 ms) with `GenAiException` error code 9 (`ErrorCode.BUSY`) and the message:

```
Request cannot be processed. Either your app is out of usage quota (try
again later) or the request is from disallowed background usage (use the
API while the app is in the foreground).
```

The app was foregrounded the entire time, so this is the quota branch.

### Data

From an evaluation harness running 93 prompts through Gemini Nano (nano-v3) on a Pixel 10 Pro XL, Android 17, genai-prompt 1.0.0-beta4 – `.` is a successful generation, `E` is BUSY:

```
run A: ....................................EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
run B: .........................................EEEEEEEEEEEEEEEEEEEEEEEEEEEEEE.EEEEEEEEEEEEE...
```

Run B started ~2.5 minutes after run A and received a fresh allowance of almost identical size, suggesting a rolling window. Pacing requests ~1 s apart and honoring `GenAiException.getRetryDelay()` avoids the wall entirely.

### Why this matters

Any batch-style usage – evaluation harnesses, prefetching several summaries, retry loops – hits this wall with no way to anticipate it. The error is retryable, and the SDK even ships `getRetryDelay()`, but neither the quota's existence, its approximate size, nor the retry-delay semantics are documented. Developers will misread mass BUSY failures as model or device breakage (we initially did).

### Requested

1. Document the per-app quota's existence and rough shape (requests per
window, replenishment behavior) in the GenAI API guides.
2. Document `ErrorCode.BUSY` and `getRetryDelay()` semantics and recommended
backoff handling.
3. Ideally: expose remaining quota (even coarsely) so apps can route to a
fallback before hitting the wall.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the GenAI API guides and the ErrorCode.BUSY and getRetryDelay() entry points mentioned in the issue. Document the approximate quota and replenishment behavior, retry-delay semantics, and recommended backoff handling; completion should cover the requested guide updates and clarify whether remaining-quota exposure is feasible.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, kotlin
Domain
ai, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.