Kong / Kong/developer.konghq.com
(ai-gateway): Create a how-to for limiting AI usage by token_cost
- Dominant language
- Ruby
- Stars
- 28
- Forks
- 121
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 313
Description
The AI Rate Limiting Advanced plugin gained a policy-based configuration system in 3.14 (with the AI Rate Limiting Policy entity) and cost-based token counting in 3.8. There is currently one how-to guide covering the custom function approach ([Enforce AI rate limits with a custom function](https://developer.konghq.com/how-to/use-custom-function-for-ai-rate-limiting/)), but no guides covering the new policy system or cost-based limiting. This issue tracks three guides to fill that gap.
**What it covers:**
- Configure `input_cost` and `output_cost` per million tokens in AI Proxy `model.options`
- Set `tokens_count_strategy: cost` in AI Rate Limiting Advanced
- Set a cost-based limit threshold (e.g. in dollars or credits)
- Validate that a cost-exceeding request returns 429
*Note on scoping* `tokens_count_strategy` appears in two places in the plugin schema: a top-level `config.tokens_count_strategy` (applies to all policies) and per-policy under `config.policies[].limits[].tokens_count_strategy` (different strategy per policy). The guide author should decide whether to cover one or both approaches, or split cost-per-policy into a separate guide.
Size: S
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the existing “Enforce AI rate limits with a custom function” how-to, then review the AI Proxy model.options and AI Rate Limiting Advanced policy configuration described in the issue. Document input_cost, output_cost, cost-based token counting, the limit threshold, and 429 validation; decide whether the top-level and per-policy strategies need one or multiple guides.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100