Kong / Kong/developer.konghq.com
[post-3.14] AI Gateway: Integrate with an OpenAI-compatible LLM provider
- Dominant language
- Ruby
- Stars
- 28
- Forks
- 121
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 313
Description
As AI Gateway is compatible with OpenAI API, users can integrate with a "dummy" OpenAI compatible provider, such as Fireworks AI, OpenRouter, SambaNova, Together AI and many others.
This requires a how-to guide covering the `provider: openai` plus `upstream_url` pattern, client SDK setup, and known limitations and possibly adding an `openai-compatible` provider to the LLM Providers.
**DoD**
- A how-to guide covering the `provider: openai` + `upstream_url` mechanism
- Plugin configuration examples for providers like Fireworks AI, OpenRouter, SambaNova, and Together AI (full list TBD)
- OpenAI Python and JS SDK client setup (overriding `base_url` / `baseURL` to point at Kong)
- Limitations section covering: token cost tracking, model name rigidity, streaming support, and provider-specific header injection
- TBD: add a dummy OpenAI-compatible provider entry to `ai-proxy.yaml` spec (to be discussed with Greg and the team)
**Size:** M to L
**Person of contact:** Greg an possibly Jack T.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.