hashicorp / hashicorp/consul

Retries do not apply to all subset when used with a ServiceSplitter

Open
#9,756 0 comments 0 reactions 0 assignees View on GitHub
theme/connect type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

When using retries with Canary deployments the retry will always be applied to the original cluster in the weighted group. It will never retry using the other clusters in the weighted group. In the instance that the canary services are all failing the retry can not be used to protect the downstream user from the problem by eventually calling one of the good endpoints in the primary group.

#### Reproduction Steps

Deploy the config required for running a canary deployment, with a retry on the `ServiceRouter`:

```
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
name: web
spec:
protocol: http

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
name: web
spec:
routes:
- destination:
service: "web"
numRetries: 20
retryOnStatusCodes: [500, 501, 502, 503]

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
name: web
spec:
defaultSubset: web-primary
subsets:
web-primary:
filter: "Service.ID contains \"web-primary\""
onlyPassing: true
web-canary:
filter: "Service.ID not contains \"web-primary\""
onlyPassing: true

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceSplitter
metadata:
name: web
spec:
splits:
- weight: 10
serviceSubset: web-canary
- weight: 90
serviceSubset: web-primary
```

When the canary is configured to fail for 100% of all requests output similar to the following can be seen.

![image](https://user-images.githubusercontent.com/773533/107647905-93b3b900-6c73-11eb-84d8-8e3fc689d64e.png)

The downstream requests show that 500 errors are being returned to the client which should not happen as the retry should allow the selection of a good endpoint from the primary subset.

This is due to Envoy only retrying endpoints from the first cluster it selects in the weighted group.'

https://github.com/envoyproxy/envoy/issues/10683

### Log Fragments

```
"dynamic_route_configs": [
{
"version_info": "00000209",
"route_config": {
"@type": "type.googleapis.com/envoy.api.v2.RouteConfiguration",
"name": "web",
"virtual_hosts": [
{
"name": "web",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "web-canary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 0
},
{
"name": "web-primary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 10000
}
],
"total_weight": 10000
},
"retry_policy": {
"retry_on": "retriable-status-codes",
"num_retries": 3,
"retriable_status_codes": [
500,
503
]
}
}
},
{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [
{
"name": "web-canary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 0
},
{
"name": "web-primary.web.default.dc1.internal.9be3fbc4-584e-5e01-62b3-b38f1352d44c.consul",
"weight": 10000
}
],
"total_weight": 10000
}
}
}
]
}
],
"validate_clusters": true
},
"last_updated": "2021-02-11T14:05:11.187Z"
},
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.