Kong / Kong/developer.konghq.com

[post-3.14] AI Gateway: Integrate with an OpenAI-compatible LLM provider

Open
#4,316 1 comment 0 reactions 1 assignee Claimed by @tomek-labuk View on GitHub
internal priority: medium
Dominant language
Ruby
Stars
28
Forks
121
Avg merge
1d 4h
Merged PRs (30d)
313

Description

As AI Gateway is compatible with OpenAI API, users can integrate with a "dummy" OpenAI compatible provider, such as Fireworks AI, OpenRouter, SambaNova, Together AI and many others.

This requires a how-to guide covering the `provider: openai` plus `upstream_url` pattern, client SDK setup, and known limitations and possibly adding an `openai-compatible` provider to the LLM Providers.

**DoD**
- A how-to guide covering the `provider: openai` + `upstream_url` mechanism
- Plugin configuration examples for providers like Fireworks AI, OpenRouter, SambaNova, and Together AI (full list TBD)
- OpenAI Python and JS SDK client setup (overriding `base_url` / `baseURL` to point at Kong)
- Limitations section covering: token cost tracking, model name rigidity, streaming support, and provider-specific header injection
- TBD: add a dummy OpenAI-compatible provider entry to `ai-proxy.yaml` spec (to be discussed with Greg and the team)

**Size:** M to L

**Person of contact:** Greg an possibly Jack T.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.