Retries do not apply to all subset when used with a ServiceSplitter
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
When using retries with Canary deployments the retry will always be applied to the original cluster in the weighted group. It will never retry using the other clusters in the weighted group. In the instance that the canary services are all failing the retry can not be used to protect the downstream user from the problem by eventually calling one of the good endpoints in the primary group.
#### Reproduction Steps
Deploy the config required for running a canary deployment, with a retry on the `ServiceRouter`:
```
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
name: web
spec:
protocol: http
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
name: web
spec:
routes:
- destination:
service: "web"
numRetries: 20
retryOnStatusCodes: [500, 501, 502, 503]
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
name: web
spec:
defaultSubset: web-primary
subsets:
web-primary:
filter: "Service.ID contains \"web-primary\""
onlyPassing: true
web-canary:
filter: "Service.ID not contains \"web-primary\""
onlyPassing: true
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceSplitter
metadata:
name: web
spec:
splits:
- weight: 10
serviceSubset: web-canary
- weight: 90
serviceSubset: web-primary
```
When the canary is configured to fail for 100% of all requests output similar to the following can be seen.

The downstream requests show that 500 errors are being returned to the client which should not happen as the retry should allow the selection of a good endpoint from the primary subset.
This is due to Envoy only retrying endpoints from the first cluster it selects in the weighted group.'
https://github.com/envoyproxy/envoy/issues/10683
### Log Fragments
```
"dynamic_route_configs": [
{
"version_info": "00000209",
"route_config": {
"@type": "type.googleapis.com/envoy.api.v2.RouteConfiguration",
"name": "web",
"virtual_hosts": [
{
"name": "web",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "web-canary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 0
},
{
"name": "web-primary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 10000
}
],
"total_weight": 10000
},
"retry_policy": {
"retry_on": "retriable-status-codes",
"num_retries": 3,
"retriable_status_codes": [
500,
503
]
}
}
},
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "web-canary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 0
},
{
"name": "web-primary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 10000
}
],
"total_weight": 10000
}
}
}
]
}
],
"validate_clusters": true
},
"last_updated": "2021-02-11T14:05:11.187Z"
},
```
Contributor guide
Assessment
This issue has not been assessed yet.