hashicorp / hashicorp/consul

Configure MaxFailures on Ingress Gateway

Open
#12,932 3 comments 0 reactions 0 assignees View on GitHub
type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Question

Hi there.

TLDR; How to configure `MaxFailures` or envoy `consecutive_5xx` of Ingress Gateway ?

I'm using consul for years and now I want to use it with Kubernetes.
I'm working with services inside and outside Kubernetes, so I must use the Ingress Gateway for the communication. The issue with Ingress Gateway (and envoy dynamic configuration) is the default consecutive 5xx configuration. This is irrelevant in our use and degrades our performance when returning 5xx. After 5 consecutive 5xx from a backend, envoy returns this message: `no healthy upstream`.

I check the consul main repository and found this issue https://github.com/hashicorp/consul/issues/11422 but this did not work (maybe because we should use `ProxyDefaults` for Ingress Gateway?).

I also checked the documentation for [`ProxyDefaults` Kubernetes YAML](https://www.consul.io/docs/connect/config-entries/proxy-defaults#config) with the link to [envoy proxy config options](https://www.consul.io/docs/connect/proxies/envoy#proxy-config-options) and saw this key `passive_health_check` but did not understand if I can and where I can add this in the proxy defaults configuration...

In my example bellow, the service returns the wanted status code `curl http://127.0.0.1:8080/status?code=500` will return a status 500 code.

### CLI Commands (consul-k8s, consul-k8s-control-plane, helm)

```
helm upgrade --install --version 0.39.0 --create-namespace -n consul --set global.name=consul -f values.yaml consul hashicorp/consul
```

### Helm Configuration

Steps to reproduce this issue, eg:

1. When running helm install with the following `values.yaml`:
```yml
global:
datacenter: 'dc1'
tls:
enabled: false
acls:
manageSystemACLs: false
image: 'hashicorp/consul:1.11.1'
imageEnvoy: 'envoyproxy/envoy-alpine:v1.20.0'
imageK8S: 'hashicorp/consul-k8s-control-plane:0.39.0'
metrics:
enabled: false
meshGateway:
enabled: false

ingressGateways:
enabled: true
defaults:
replicas: 1
service:
ports:
- port: 30000
nodePort: 30000
gateways:
- name: ingress-gateway
service:
type: NodePort
terminatingGateways:
enabled: false
connectInject:
enabled: true
default: true
replicas: 1

metrics:
defaultEnabled: false
server:
enabled: true
replicas: 1
externalServers:
enabled: false
client:
grpc: true
extraConfig: |
{"advertise_reconnect_timeout": "1h"}
controller:
enabled: true
dns:
enabled: true
syncCatalog:
enabled: false
```
2. The `ingress-gateway.yaml` manifest
```yml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
config:
protocol: 'http'
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: IngressGateway
metadata:
name: ingress-gateway
spec:
listeners:
- port: 30000
protocol: http
services:
- name: statuscode
```
3. The `status-code.yaml` manifest
```yaml
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: statuscode
name: statuscode
spec:
ports:
- port: 8080
selector:
app.kubernetes.io/name: statuscode
status:
loadBalancer: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: statuscode
name: statuscode
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: statuscode
strategy: {}
template:
metadata:
labels:
app.kubernetes.io/name: statuscode
annotations:
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/connect-service': 'statuscode'
'consul.hashicorp.com/connect-port': '8080'
spec:
containers:
- image: dotnsf/statuscode_generator
name: statuscode
ports:
- containerPort: 8080
resources:
requests: { memory: '256Mi' }
limits: { memory: '1Gi' }
readinessProbe:
httpGet: { path: /, port: 8080 }
initialDelaySeconds: 20
periodSeconds: 10
livenessProbe:
httpGet: { path: /, port: 8080 }
initialDelaySeconds: 20
periodSeconds: 10
restartPolicy: Always
status: {}
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
name: statuscode
spec:
protocol: http
upstreamConfig:
defaults:
passiveHealthCheck:
maxFailures: 400
interval: '1s'
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
name: statuscode
spec:
destination:
name: statuscode
sources:
- name: 'ingress-gateway'
action: allow
```

### Logs

When I check the ingress-gateway config dump (http://127.0.0.1:19000/config_dump), where is what I found, outlier_detection still empty:
```sh
kubectl port-forward -n consul consul-ingress-gateway-5d9dc99bbd-shw5r 19000
curl http://127.0.0.1:19000/config_dump
```
```json
{
"version_info": "ea72e0ee8944a6e8a3d74828cd48c542dfbb8c70b59be52e83d4d0159f18e4af",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "statuscode.default.dc1.internal.0ad86cd7-4cfa-e946-8f67-b920776a6757.consul",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {},
"resource_api_version": "V3"
}
},
"connect_timeout": "5s",
"circuit_breakers": {},
"outlier_detection": {},
```

You can access the Ingress Gateway on your computer on port 30000, send 6 times this request:
```sh
curl -H 'Host: statuscode.ingress.consul' http://:30000/status?code=500
```

### Current understanding and Expected behavior

### Environment details

- `consul-k8s` version: 0.39.0
- `values.yaml` used to deploy the helm chart: :arrow_up:

Additionally, please provide details regarding the Kubernetes Infrastructure, as shown below:
- Kubernetes version: minikube v1.23.2

### Additional Context

- [Consul Doc: Envoy Integration](https://www.consul.io/docs/connect/proxies/envoy#proxy-config-options)
- [Consul Doc: Service Defaults](https://www.consul.io/docs/connect/config-entries/service-defaults)
- [Consul Doc: Proxy Defaults](https://www.consul.io/docs/connect/config-entries/proxy-defaults)
- [Envoy Doc: OutlierDetection](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/outlier_detection.proto)
- [Consul issue: Add additional Envoy consecutive_5xx outlier ejection configuration #11422](https://github.com/hashicorp/consul/issues/11422)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.