Kong / Kong/developer.konghq.com

AI Cookbooks: Recipe for dynamic model selection with AI Gateway

Open
#5,471 0 comments 0 reactions 1 assignee Claimed by @tomek-labuk View on GitHub
ai-cookbooks internal
Dominant language
Ruby
Stars
28
Forks
121
Avg merge
1d 4h
Merged PRs (30d)
313

Description

Help documenting a pattern for routing LLM requests to different models based on content. A "model selection" route uses the Prompt Decorator plugin to ask a cheap LLM to classify the incoming query and return a model name. DataKit then rewrites the request body to set the target model before proxying to the real upstream. Semantic Cache sits on the final routes to reduce token cost.

A "model selection" route runs first. It uses the Prompt Decorator plugin to append a classifier instruction to the request and sends it to o3-mini. The response is a single model name. DataKit reads that response, rewrites the `model` field in the request body, and routes the modified request to the appropriate upstream. Semantic Cache sits on the model routes to avoid redundant calls.

```yaml
plugins:
- name: ai-prompt-decorator
config:
prompts:
append:
- role: user
content: |-
Based on the user's question, select the most appropriate model to answer it. The options are:
- gpt-4o: best for complex questions and nuanced answers
- o3-mini: best for straightforward questions and concise answers
Only respond with the model name, either "gpt-4o" or "o3-mini".
```

DataKit then calls the selection route, extracts the model name from the response, and patches the outgoing request before forwarding to the real LLM route.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.