Azure-Samples / Azure-Samples/AI-Gateway
Request to add Enterprise AI Gateway sample (Terraform, integrated LLM + MCP governance)
- Dominant language
- Jupyter Notebook
- Stars
- 986
- Forks
- 513
- Avg merge
- 12m
- Merged PRs (30d)
- 1
Description
## What I'm asking
I built an [Enterprise AI Gateway](https://github.com/nicksangeorge/enterprise-ai-gateway) reference implementation that pulls together several capabilities this repo covers as standalone labs. I'd like to move it into AI-Gateway as a new lab so it lives alongside the existing samples.
The repo is live and working today. You can review the full source, docs, and test suite there.
## Why it belongs here
The individual labs are the right place to learn each capability on its own. What's missing is a sample that shows how they fit together in an enterprise setup: three teams sharing one APIM gateway with different token quotas, multi-region failover with circuit breakers, MCP tools governed through the same gateway, and a chargeback dashboard that breaks cost down by team.
This sample composes patterns from `backend-pool-load-balancing`, `token-rate-limiting`, `token-metrics-emitting`, `finops-framework`, and the MCP labs into one deployment. It doesn't replace any of them.
## What the sample includes
- **Full-stack Terraform**: APIM, Foundry (2 regions), App Insights, Log Analytics, API Center. One `terraform apply`.
- **Per-team token quotas**: Three products (Alpha 50K TPM, Beta 20K TPM, Gamma 500 TPM) with `llm-token-limit` enforcement.
- **Multi-region failover**: Backend pool with priority routing and circuit breakers that read `Retry-After` headers.
- **Token chargeback**: `llm-emit-token-metric` with team/model/region dimensions, KQL queries, and an Azure Monitor workbook.
- **MCP tool governance**: MCP servers registered in APIM with per-session rate limits, correlation IDs, and audit logging.
- **API Center discovery**: APIM APIs and MCP servers synced to API Center as a searchable catalog.
- **Automated test suite**: 15 tests (7 LLM gateway, 4 MCP governance, 4 MCP rate limit) runnable outside the notebook.
## Gaps it fills
- No existing lab deploys the full stack with Terraform (only `backend-pool-load-balancing-tf` exists, and it covers pools only).
- No existing lab combines LLM gateway and MCP governance in a single deployment.
- No existing lab ships a runnable test suite alongside notebook validation.
- `zero-to-production` is the closest match but skips chargeback dashboards and MCP governance.
## Proposed structure
Following existing lab conventions: `README.MD` with frontmatter, Jupyter notebook walkthrough, `clean-up-resources.ipynb`, Terraform files, APIM policy XMLs, test scripts, and `result.png`.
Proposed path: `labs/enterprise-ai-gateway/`
## One question
The sample is Terraform only. Happy to add Bicep in a follow-up if that's preferred. Wanted to check before investing the effort.
Contributor guide
Assessment
This issue has not been assessed yet.