dapr / dapr/components-contrib

Azure OpenAI binding: Enable support for multiple endpoints

Open
#3,381 1 comment 2 reactions 0 assignees View on GitHub
kind/enhancement P1 pinned
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## Describe the feature

The current [Azure OpenAI binding](https://docs.dapr.io/reference/components-reference/supported-bindings/openai/) takes in configuration for connecting to a single Azure OpenAI endpoint.

In an number of scenarios it is useful to be able to work with a number of Azure OpenAI endpoints.

### Scenario 1 - fail-over
For high-volume usage, customers may purchase a [Provisioned Throughput Unit](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput)(PTU). In this scenario, the PTU capacity isn't always sufficient for peak-load and a customer might want to send a request to the PTU first and then re-send to a Pay-As-You-Go (PAYG) endpoint if the PTU endpoint returns a 429 response.

### Scenario 2 - round-robin
The limits for Azure OpenAI are [per-region](https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits#regional-quota-limits) and customers may set up multiple PAYG endpoints across regions and want to distribute requests between them

### Proposal

Sometimes customers with either of the above requirements will set up a gateway in front of the Azure OpenAI endpoints and have that handle the load distribution, but in other cases they come back to the application code to add these capabilities in as the usage scales up.

The proposal is to update the Azure OpenAI binding to allow multiple endpoints to be configured along with a distribution mode (`failover` or `round-robin`).

## Release Note

RELEASE NOTE: **ADD** Enable multiple endpoints to be configured in Azure OpenAI binding.

Contributor guide

Open the contributing guide

Research direction

Start with the Azure OpenAI binding and its existing single-endpoint configuration, then trace how requests are dispatched. Define and test failover and round-robin behavior, including the 429 case, and update the binding documentation; done means multiple endpoints and both distribution modes work.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
ai, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.