Azure-Samples / Azure-Samples/AI-Gateway

OrcaRouter provider support for AI Gateway Labs

Open
#394 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
986
Forks
513
Avg merge
12m
Merged PRs (30d)
1

Description

## Proposal: optional OrcaRouter provider for AI Gateway Labs

AI Gateway Labs is a genuinely useful resource for developers and platform teams putting production AI applications on Azure API Management and Microsoft Foundry. The hands-on notebooks turn hard operational problems, such as distributing load across model endpoints, routing by model and version, semantic caching, and FinOps budget controls, into reproducible labs with Bicep templates and APIM policies.

Because the gateway's value rests on giving teams more choice over the models behind their applications, each additional well-behaved upstream provider makes those labs more relevant. An optional OrcaRouter backend would fit naturally: an OpenAI-compatible endpoint that users could point the model-routing and backend-pool labs at, exercising the same APIM policies against a different multi-model provider.

I'm an engineer on the OrcaRouter team, and I'd like to propose adding [OrcaRouter](https://www.orcarouter.ai) as an optional provider for AI Gateway Labs. This would be purely additive; no existing provider, lab, or APIM policy would be replaced or changed, and teams happy with their current backends can simply ignore it.

A few OrcaRouter capabilities map directly to what these labs teach:

- **One OpenAI-compatible endpoint across many models**, including chat, reasoning, image, and video, matching the backend shape the gateway already expects.
- **Automatic model routing and provider failover**, giving the model-routing and backend-pool-load-balancing labs another realistic scenario.
- **Usage tracking and budgets** per key and team, aligning with the FinOps and token-rate-limiting labs central to this project.

### Integration point

OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, matching how this repo already treats backends (its own Mock Server tool is an OpenAI API mock for local development). The expected integration would be documenting OrcaRouter as an additional upstream backend for the model-routing and backend-pool labs, with the key supplied through the usual APIM policy secret mechanism. This is a proposal only; no code has been written or tested here.

OrcaRouter is already used in the open-source ecosystem as an OpenAI-compatible provider; projects such as RAGFlow, Dify, and goose plug it in alongside other model providers.

### Transparency

OrcaRouter runs an optional open-source partner program in which approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participation is not a prerequisite for integration, and I'm happy to follow this project's disclosure and governance requirements, including the Microsoft CLA process for any eventual contribution.

### Next steps

See examples of other open-source integrations at https://www.orcarouter.ai/built-with. If the maintainers see value in this, I'd welcome your thoughts on scope, such as a dedicated lab notebook, an addition to the model-routing lab, or documentation alone, and I'd gladly submit an implementation PR once the direction is approved.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the model-routing and backend-pool labs, their APIM policies, and the Mock Server tool to understand how upstream backends are represented. Confirm the maintainers' preferred scope before work begins; done would mean an approved optional OrcaRouter integration that leaves existing labs, providers, and policies unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, jupyter-notebook
Domain
api, cloud, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.