Azure-Samples / Azure-Samples/AI-Gateway
OrcaRouter provider support for AI Gateway Labs
- Dominant language
- Jupyter Notebook
- Stars
- 986
- Forks
- 513
- Avg merge
- 12m
- Merged PRs (30d)
- 1
Description
## Proposal: optional OrcaRouter provider for AI Gateway Labs
AI Gateway Labs is a genuinely useful resource for developers and platform teams putting production AI applications on Azure API Management and Microsoft Foundry. The hands-on notebooks turn hard operational problems, such as distributing load across model endpoints, routing by model and version, semantic caching, and FinOps budget controls, into reproducible labs with Bicep templates and APIM policies.
Because the gateway's value rests on giving teams more choice over the models behind their applications, each additional well-behaved upstream provider makes those labs more relevant. An optional OrcaRouter backend would fit naturally: an OpenAI-compatible endpoint that users could point the model-routing and backend-pool labs at, exercising the same APIM policies against a different multi-model provider.
I'm an engineer on the OrcaRouter team, and I'd like to propose adding [OrcaRouter](https://www.orcarouter.ai) as an optional provider for AI Gateway Labs. This would be purely additive; no existing provider, lab, or APIM policy would be replaced or changed, and teams happy with their current backends can simply ignore it.
A few OrcaRouter capabilities map directly to what these labs teach:
- **One OpenAI-compatible endpoint across many models**, including chat, reasoning, image, and video, matching the backend shape the gateway already expects.
- **Automatic model routing and provider failover**, giving the model-routing and backend-pool-load-balancing labs another realistic scenario.
- **Usage tracking and budgets** per key and team, aligning with the FinOps and token-rate-limiting labs central to this project.
### Integration point
OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, matching how this repo already treats backends (its own Mock Server tool is an OpenAI API mock for local development). The expected integration would be documenting OrcaRouter as an additional upstream backend for the model-routing and backend-pool labs, with the key supplied through the usual APIM policy secret mechanism. This is a proposal only; no code has been written or tested here.
OrcaRouter is already used in the open-source ecosystem as an OpenAI-compatible provider; projects such as RAGFlow, Dify, and goose plug it in alongside other model providers.
### Transparency
OrcaRouter runs an optional open-source partner program in which approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participation is not a prerequisite for integration, and I'm happy to follow this project's disclosure and governance requirements, including the Microsoft CLA process for any eventual contribution.
### Next steps
See examples of other open-source integrations at https://www.orcarouter.ai/built-with. If the maintainers see value in this, I'd welcome your thoughts on scope, such as a dedicated lab notebook, an addition to the model-routing lab, or documentation alone, and I'd gladly submit an implementation PR once the direction is approved.
Contributor guide
Research direction
Start by reviewing the model-routing and backend-pool labs, their APIM policies, and the Mock Server tool to understand how upstream backends are represented. Confirm the maintainers' preferred scope before work begins; done would mean an approved optional OrcaRouter integration that leaves existing labs, providers, and policies unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, jupyter-notebook
- Domain
- api, cloud, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100