dapr / dapr/components-contrib
Azure OpenAI binding: Enable support for multiple endpoints
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
## Describe the feature
The current [Azure OpenAI binding](https://docs.dapr.io/reference/components-reference/supported-bindings/openai/) takes in configuration for connecting to a single Azure OpenAI endpoint.
In an number of scenarios it is useful to be able to work with a number of Azure OpenAI endpoints.
### Scenario 1 - fail-over
For high-volume usage, customers may purchase a [Provisioned Throughput Unit](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput)(PTU). In this scenario, the PTU capacity isn't always sufficient for peak-load and a customer might want to send a request to the PTU first and then re-send to a Pay-As-You-Go (PAYG) endpoint if the PTU endpoint returns a 429 response.
### Scenario 2 - round-robin
The limits for Azure OpenAI are [per-region](https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits#regional-quota-limits) and customers may set up multiple PAYG endpoints across regions and want to distribute requests between them
### Proposal
Sometimes customers with either of the above requirements will set up a gateway in front of the Azure OpenAI endpoints and have that handle the load distribution, but in other cases they come back to the application code to add these capabilities in as the usage scales up.
The proposal is to update the Azure OpenAI binding to allow multiple endpoints to be configured along with a distribution mode (`failover` or `round-robin`).
## Release Note
RELEASE NOTE: **ADD** Enable multiple endpoints to be configured in Azure OpenAI binding.
Contributor guide
Research direction
Start with the Azure OpenAI binding and its existing single-endpoint configuration, then trace how requests are dispatched. Define and test failover and round-robin behavior, including the 429 case, and update the binding documentation; done means multiple endpoints and both distribution modes work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- ai, backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100