Gateway with TCPRoute loses connection to backend pod when backend pod restarts

*Description*:
> There is k8s single node cluster with a Fluentbit pod which servers as a syslog (over TLS) server for external clients. Clients connect to it via Envoy Gateway and TCPRoute. The issue is that when Fluentbit pod gets recreated, it stops receiving traffic from clients for around 15 minutes. The traffic still reaches the node, as I can see from tcpdump, but it is not forwarded to Fluentbit. After 15 minutes the Gateway resets TCP session, client re-establishes and everything gets back to normal.
Also noticed that the issue doesn't happen if messages are arriving from clients while the pod restarts. E.g. it takes around 15 seconds for pod to restart, and if a client sends a message every 5-10 seconds, then everything is fine. But if the rate is like 1 message in 30 seconds, then Envoy stops forwarding it.

*Repro steps*:
> Deploy Envoy gateway with this guide https://gateway.envoyproxy.io/docs/install/install-yaml/
Deploy MetalLB: https://metallb.universe.tf/installation/
Create IPPool and configure MetalLB in L2 mode.
Deploy FluentBit and create a ClusterIP service for it.
```
$ k describe svc fbit
Name: fbit
Namespace: default
Labels: app=fbit
Annotations:
Selector: k8s-app=fluent-bit
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP:
IPs:
Port: 5190-5190 5190/TCP
TargetPort: 5190/TCP
Endpoints: 10.0.0.148:5190
Session Affinity: None
Events:
```
> Create Gateway and TCPRoute.
```
$ k get gatewayclass -o yaml
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1","kind":"GatewayClass","metadata":{"annotations":{},"name":"eg"},"spec":{"controllerName":"gateway.envoyproxy.io/gatewayclass-controller"}}
creationTimestamp: "2024-12-02T12:57:37Z"
finalizers:
- gateway-exists-finalizer.gateway.networking.k8s.io
generation: 1
name: eg
resourceVersion: "4709"
uid: 536a2915-d7e0-4100-896e-e7418ff36c57
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
status:
conditions:
- lastTransitionTime: "2024-12-02T12:57:37Z"
message: Valid GatewayClass
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
kind: List
metadata:
resourceVersion: ""

$ k get gateway -o yaml
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1","kind":"Gateway","metadata":{"annotations":{},"name":"tcp","namespace":"default"},"spec":{"gatewayClassName":"eg","listeners":[{"allowedRoutes":{"kinds":[{"kind":"TCPRoute"}],"namespaces":{"from":"All"}},"name":"tcpterm","port":6514,"protocol":"TCP"}]}}
creationTimestamp: "2024-12-03T05:48:48Z"
generation: 1
name: tcp
namespace: default
resourceVersion: "145420"
uid: 0bd76b74-5fe5-474d-b64a-a6e448402b7a
spec:
gatewayClassName: eg
listeners:
- allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: TCPRoute
namespaces:
from: All
name: tcpterm
port: 6514
protocol: TCP
status:
addresses:
- type: IPAddress
value:
conditions:
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: The Gateway has been scheduled by Envoy Gateway
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: Address assigned to the Gateway, 1/1 envoy replicas available
observedGeneration: 1
reason: Programmed
status: "True"
type: Programmed
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: Sending translated listener configuration to the data plane
observedGeneration: 1
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: Listener has been successfully translated
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: Listener references have been resolved
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: tcpterm
supportedKinds:
- group: gateway.networking.k8s.io
kind: TCPRoute
kind: List
metadata:
resourceVersion: ""

$ k get tcproute -o yaml
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"TCPRoute","metadata":{"annotations":{},"name":"tcp-fbit","namespace":"default"},"spec":{"parentRefs":[{"name":"tcp","sectionName":"tcpterm"}],"rules":[{"backendRefs":[{"name":"fbit","port":5190}]}]}}
creationTimestamp: "2024-12-03T05:49:30Z"
generation: 1
name: tcp-fbit
namespace: default
resourceVersion: "145419"
uid: fd1cf2d7-26c6-4e08-b522-d54e948fe7ce
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: tcp
sectionName: tcpterm
rules:
- backendRefs:
- group: ""
kind: Service
name: fbit
port: 5190
weight: 1
status:
parents:
- conditions:
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: Route is accepted
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-12-03T05:49:30Z"
message: Resolved all the Object references for the Route
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: tcp
sectionName: tcpterm
kind: List
metadata:
resourceVersion: ""
```
> Assign Gateway IP to local server interface (otherwise MetalLB didn't respond to ARP requests).
Configure syslog client.
Restart the fluentbit pod.

*Environment*:
> Ubuntu 22.04
k8s v1.30.7
envoyproxy/gateway:v1.2.3

*Logs*:
> Pod killed at 13:31:05 and came back at 13:31:17.
Fluentbit logs:
```
$ k logs fluent-bit-2k2vr | head -30
Defaulted container "fluent-bit" out of: fluent-bit, sidecar
Fluent Bit v3.2.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
[2024/12/03 13:31:18] [ info] [input:syslog:syslog.0] initializing
[2024/12/03 13:31:18] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2024/12/03 13:31:18] [ info] [in_syslog] TCP server binding 0.0.0.0:5190
[2024/12/03 13:31:18] [ info] [output:stdout:stdout.0] worker #0 started
[0] client: [[1733233643.000000000, {}], {"pri"=>"188" -> Here comes the first message.
```
> 1733233643 -> 13:47:23, so it took 16 minutes to restore.
The env was configured so that the client was sending packets every 15 seconds:
```
13:32:04.538539 CLIENTMAC > SERVERMAC, ethertype IPv4 (0x0800), length 225: CLIENT_IP.56939 > SERVER_IP.6514: Flags [P.], seq 1228839153:1228839312, ack 3234245467, win 7714, options [nop,nop,TS val 3420375641 ecr 86463925], length 159
13:32:04.538540 CLIENTMAC > SERVERMAC, ethertype IPv4 (0x0800), length 220: CLIENT_IP.56939 > SERVER_IP.6514: Flags [P.], seq 159:313, ack 1, win 7714, options [nop,nop,TS val 3420375641 ecr 86463925], length 154
13:32:04.538540 CLIENTMAC > SERVERMAC, ethertype IPv4 (0x0800), length 272: CLIENT_IP.56939 > SERVER_IP.6514: Flags [P.], seq 313:519, ack 1, win 7714, options [nop,nop,TS val 3420375641 ecr 86463925], length 206
13:32:04.538840 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 159, win 501, options [nop,nop,TS val 86479205 ecr 3420375641], length 0
13:32:04.538871 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 313, win 501, options [nop,nop,TS val 86479205 ecr 3420375641], length 0
13:32:04.538882 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 519, win 501, options [nop,nop,TS val 86479205 ecr 3420375641], length 0
13:32:20.089850 CLIENTMAC > SERVERMAC, ethertype IPv4 (0x0800), length 220: CLIENT_IP.56939 > SERVER_IP.6514: Flags [P.], seq 995:1149, ack 1, win 7714, options [nop,nop,TS val 3420375673 ecr 86479515], length 154
13:32:20.090226 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 995, win 501, options [nop,nop,TS val 86494756 ecr 3420375673], length 0
13:32:20.090274 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 1149, win 501, options [nop,nop,TS val 86494756 ecr 3420375673], length 0
13:32:20.090310 CLIENTMAC > SERVERMAC, ethertype IPv4 (0x0800), length 272: CLIENT_IP.56939 > SERVER_IP.6514: Flags [P.], seq 1149:1355, ack 1, win 7714, options [nop,nop,TS val 3420375673 ecr 86479515], length 206
13:32:20.090385 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 1355, win 501, options [nop,nop,TS val 86494757 ecr 3420375673], length 0
13:32:20.398103 CLIENTMAC > SERVERMAC, ethertype IPv4 (0x0800), length 227: CLIENT_IP.56939 > SERVER_IP.6514: Flags [P.], seq 1355:1516, ack 1, win 7714, options [nop,nop,TS val 3420375673 ecr 86494757], length 161
13:32:20.398346 SERVERMAC > CLIENTMAC, ethertype IPv4 (0x0800), length 66: SERVER_IP.6514 > CLIENT_IP.56939: Flags [.], ack 1516, win 501, options [nop,nop,TS val 86495064 ecr 3420375673], length 0
```
> Envoy gateway log
```
{"start_time":"2024-12-03T12:59:29.509Z","method":"-","x-envoy-origin-path":"-","protocol":"-","response_code":"0","response_flags":"-","response_code_details":"-","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"29797","bytes_sent":"2074","duration":"1019939","x-envoy-upstream-service-time":"-","x-forwarded-for":"-","user-agent":"-","x-request-id":"-",":authority":"-","upstream_host":"10.0.0.112:5190","upstream_cluster":"tcproute/default/tcp-fbit/rule/-1","upstream_local_address":"10.0.0.5:47848","downstream_local_address":"10.0.0.5:6514","downstream_remote_address":"SERVER_IP:56913","requested_server_name":"-","route_name":"-"}
{"start_time":"2024-12-03T13:17:00.037Z","method":"-","x-envoy-origin-path":"-","protocol":"-","response_code":"0","response_flags":"-","response_code_details":"-","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"74625","bytes_sent":"2074","duration":"1807401","x-envoy-upstream-service-time":"-","x-forwarded-for":"-","user-agent":"-","x-request-id":"-",":authority":"-","upstream_host":"10.0.0.66:5190","upstream_cluster":"tcproute/default/tcp-fbit/rule/-1","upstream_local_address":"10.0.0.5:55434","downstream_local_address":"10.0.0.5:6514","downstream_remote_address":"SERVER_IP:56939","requested_server_name":"-","route_name":"-"}
{"start_time":"2024-12-03T13:47:23.013Z","method":"-","x-envoy-origin-path":"-","protocol":"-","response_code":"0","response_flags":"-","response_code_details":"-","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"354166","bytes_sent":"2074","duration":"5757355","x-envoy-upstream-service-time":"-","x-forwarded-for":"-","user-agent":"-","x-request-id":"-",":authority":"-","upstream_host":"10.0.0.148:5190","upstream_cluster":"tcproute/default/tcp-fbit/rule/-1","upstream_local_address":"10.0.0.5:35442","downstream_local_address":"10.0.0.5:6514","downstream_remote_address":"CLIENT_IP_IP:56989","requested_server_name":"-","route_name":"-"}

```