envoyproxy / envoyproxy/gateway

Envoy doesn't respect EndpointSlice terminating: true pod status

Open
#6,277 9 comments 10 reactions 0 assignees View on GitHub
stale triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:
During upstream pod replacement with GRPC stream opened i'm receiving GRPC status code 13, i'm expecting status code 14.

```
gRPC error: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Received RST_STREAM with error code 0"
debug_error_string = "UNKNOWN:Error received from peer ipv4::443 {grpc_status:13, grpc_message:"Received RST_STREAM with error code 0"}"
```

I did some research and saw that envoy deletes cluster endpoint(eg pod IP) from list only when pod is deleted from endpoint slice, when it's still there but with status `terminating: true` nothing happens at all, for example AWS ALB controller starts to deregister pod from target group when pod in terminating status.

I was expecting that envoy respects `terminating: true` status of pod in endpoint slice and will start to drain connections to this pod.

*Repro steps*:
1. Open grpc stream (any dummy grpc server and client should suite)
2. Restart upstream pod
3. Observe an error.

Envoy config below:
```
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name:
namespace: envoy-gateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name:
timeout:
http:
connectionIdleTimeout: 30s
requestTimeout: 0s
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
name:
namespace: envoy-gateway-system
spec:
enableProxyProtocol: true
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name:
timeout:
http:
idleTimeout: 120s
tcp:
idleTimeout: 120s
---
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
name: -443
namespace:
spec:
hostnames:
- example.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name:
namespace: envoy-gateway-system
port: 443
rules:
- backendRefs:
- group: ''
kind: Service
name: -stable
port: 443
weight: 100
```

*Environment*:
```
gateway version: v1.4.0
envoy version: v1.34.1
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.