Kong / Kong/developer.konghq.com

(ai-gateway): Create a how-to for limiting AI usage by token_cost

Open
#4,682 0 comments 0 reactions 0 assignees View on GitHub
internal product:ai-gateway
Dominant language
Ruby
Stars
28
Forks
121
Avg merge
1d 4h
Merged PRs (30d)
313

Description

The AI Rate Limiting Advanced plugin gained a policy-based configuration system in 3.14 (with the AI Rate Limiting Policy entity) and cost-based token counting in 3.8. There is currently one how-to guide covering the custom function approach ([Enforce AI rate limits with a custom function](https://developer.konghq.com/how-to/use-custom-function-for-ai-rate-limiting/)), but no guides covering the new policy system or cost-based limiting. This issue tracks three guides to fill that gap.

**What it covers:**
- Configure `input_cost` and `output_cost` per million tokens in AI Proxy `model.options`
- Set `tokens_count_strategy: cost` in AI Rate Limiting Advanced
- Set a cost-based limit threshold (e.g. in dollars or credits)
- Validate that a cost-exceeding request returns 429

*Note on scoping* `tokens_count_strategy` appears in two places in the plugin schema: a top-level `config.tokens_count_strategy` (applies to all policies) and per-policy under `config.policies[].limits[].tokens_count_strategy` (different strategy per policy). The guide author should decide whether to cover one or both approaches, or split cost-per-policy into a separate guide.

Size: S

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the existing “Enforce AI rate limits with a custom function” how-to, then review the AI Proxy model.options and AI Rate Limiting Advanced policy configuration described in the issue. Document input_cost, output_cost, cost-based token counting, the limit threshold, and 429 validation; decide whether the top-level and per-policy strategies need one or multiple guides.

Written by the indexing model from the issue text.

Assessment

Domain
ai, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.