Support fallback to different backend cluster based on response status code from the primary cluster
Open
area/retry
enhancement
no stalebot
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
*Title*: *Support fallback to different backend cluster based on response status code from the primary cluster*
*Description*:
Currently envoy retry is based on priority of the clusters and health status, for AI inference workload we want to fallback to different endpoints(cluster) based on the status code of the response from primary destination.
For example if the provisioned throughput endpoint does not enough capacity and returns 429, we want to fallback to on-demand endpoint to continue processing the message before returning the response to user.
[optional *Relevant Links*:]
>Any extra documentation required to understand the issue.
Contributor guide
Assessment
This issue has not been assessed yet.