googlesamples / googlesamples/mlkit
Document AICore's per-app inference quota – sustained generation returns GenAiException BUSY (error code 9) after ~40 requests
- Dominant language
- Kotlin
- Stars
- 4.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
### Description
Sustained back-to-back inference through the Prompt API hits a rolling per-app quota that is not documented anywhere. After roughly 36–41 consecutive generations (~1.5 s each, so about a minute of sustained load), every subsequent request fails fast (~15–25 ms) with `GenAiException` error code 9 (`ErrorCode.BUSY`) and the message:
```
Request cannot be processed. Either your app is out of usage quota (try
again later) or the request is from disallowed background usage (use the
API while the app is in the foreground).
```
The app was foregrounded the entire time, so this is the quota branch.
### Data
From an evaluation harness running 93 prompts through Gemini Nano (nano-v3) on a Pixel 10 Pro XL, Android 17, genai-prompt 1.0.0-beta4 – `.` is a successful generation, `E` is BUSY:
```
run A: ....................................EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
run B: .........................................EEEEEEEEEEEEEEEEEEEEEEEEEEEEEE.EEEEEEEEEEEEE...
```
Run B started ~2.5 minutes after run A and received a fresh allowance of almost identical size, suggesting a rolling window. Pacing requests ~1 s apart and honoring `GenAiException.getRetryDelay()` avoids the wall entirely.
### Why this matters
Any batch-style usage – evaluation harnesses, prefetching several summaries, retry loops – hits this wall with no way to anticipate it. The error is retryable, and the SDK even ships `getRetryDelay()`, but neither the quota's existence, its approximate size, nor the retry-delay semantics are documented. Developers will misread mass BUSY failures as model or device breakage (we initially did).
### Requested
1. Document the per-app quota's existence and rough shape (requests per
window, replenishment behavior) in the GenAI API guides.
2. Document `ErrorCode.BUSY` and `getRetryDelay()` semantics and recommended
backoff handling.
3. Ideally: expose remaining quota (even coarsely) so apps can route to a
fallback before hitting the wall.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the GenAI API guides and the ErrorCode.BUSY and getRetryDelay() entry points mentioned in the issue. Document the approximate quota and replenishment behavior, retry-delay semantics, and recommended backoff handling; completion should cover the requested guide updates and clarify whether remaining-quota exposure is feasible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, kotlin
- Domain
- ai, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100