Azure-Samples / Azure-Samples/AI-Gateway

Latency routing for AOAI in APIM

Open
#175 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
986
Forks
513
Avg merge
12m
Merged PRs (30d)
1

Description

When AOAI request takes longer than specified amount of time use 'forward request' policy to reroute to a different AOAI endpoint.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Azure API Management policy examples or entry points for AOAI requests and review how the 'forward request' policy targets alternate AOAI endpoints. Done means requests exceeding a configured latency threshold are routed to the different endpoint, with the threshold and fallback behavior demonstrated in the relevant example or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
api, cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.