Azure-Samples / Azure-Samples/AI-Gateway
Latency routing for AOAI in APIM
Open
- Dominant language
- Jupyter Notebook
- Stars
- 986
- Forks
- 513
- Avg merge
- 12m
- Merged PRs (30d)
- 1
Description
When AOAI request takes longer than specified amount of time use 'forward request' policy to reroute to a different AOAI endpoint.
Contributor guide
Research direction
Start by locating the Azure API Management policy examples or entry points for AOAI requests and review how the 'forward request' policy targets alternate AOAI endpoints. Done means requests exceeding a configured latency threshold are routed to the different endpoint, with the threshold and fallback behavior demonstrated in the relevant example or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- api, cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100