Azure-Samples / Azure-Samples/AI-Gateway

Request to add Enterprise AI Gateway sample (Terraform, integrated LLM + MCP governance)

Open
#304 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
986
Forks
513
Avg merge
12m
Merged PRs (30d)
1

Description

## What I'm asking

I built an [Enterprise AI Gateway](https://github.com/nicksangeorge/enterprise-ai-gateway) reference implementation that pulls together several capabilities this repo covers as standalone labs. I'd like to move it into AI-Gateway as a new lab so it lives alongside the existing samples.

The repo is live and working today. You can review the full source, docs, and test suite there.

## Why it belongs here

The individual labs are the right place to learn each capability on its own. What's missing is a sample that shows how they fit together in an enterprise setup: three teams sharing one APIM gateway with different token quotas, multi-region failover with circuit breakers, MCP tools governed through the same gateway, and a chargeback dashboard that breaks cost down by team.

This sample composes patterns from `backend-pool-load-balancing`, `token-rate-limiting`, `token-metrics-emitting`, `finops-framework`, and the MCP labs into one deployment. It doesn't replace any of them.

## What the sample includes

- **Full-stack Terraform**: APIM, Foundry (2 regions), App Insights, Log Analytics, API Center. One `terraform apply`.
- **Per-team token quotas**: Three products (Alpha 50K TPM, Beta 20K TPM, Gamma 500 TPM) with `llm-token-limit` enforcement.
- **Multi-region failover**: Backend pool with priority routing and circuit breakers that read `Retry-After` headers.
- **Token chargeback**: `llm-emit-token-metric` with team/model/region dimensions, KQL queries, and an Azure Monitor workbook.
- **MCP tool governance**: MCP servers registered in APIM with per-session rate limits, correlation IDs, and audit logging.
- **API Center discovery**: APIM APIs and MCP servers synced to API Center as a searchable catalog.
- **Automated test suite**: 15 tests (7 LLM gateway, 4 MCP governance, 4 MCP rate limit) runnable outside the notebook.

## Gaps it fills

- No existing lab deploys the full stack with Terraform (only `backend-pool-load-balancing-tf` exists, and it covers pools only).
- No existing lab combines LLM gateway and MCP governance in a single deployment.
- No existing lab ships a runnable test suite alongside notebook validation.
- `zero-to-production` is the closest match but skips chargeback dashboards and MCP governance.

## Proposed structure

Following existing lab conventions: `README.MD` with frontmatter, Jupyter notebook walkthrough, `clean-up-resources.ipynb`, Terraform files, APIM policy XMLs, test scripts, and `result.png`.

Proposed path: `labs/enterprise-ai-gateway/`

## One question

The sample is Terraform only. Happy to add Bicep in a follow-up if that's preferred. Wanted to check before investing the effort.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.