TLS termation with AWS NLB doesn't work

*Description*:

I'm trying to get the envoy gateway up and running for a https endpoint. But unfortunately it always fails. The connection is always reset.

```
curl -kvv -L https://secure.REDACTED.com
09:34:15.316000 [0-0] * Host secure.REDACTED.com:443 was resolved.
09:34:15.321000 [0-0] * IPv6: (none)
09:34:15.323000 [0-0] * IPv4: REDACTED.166, REDACTED.11
09:34:15.324000 [0-0] * [HTTPS-CONNECT] adding wanted h2
09:34:15.326000 [0-0] * [HTTPS-CONNECT] added
09:34:15.328000 [0-0] * [HTTPS-CONNECT] connect, init
09:34:15.330000 [0-0] * Trying REDACTED.166:443...
09:34:15.332000 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
09:34:15.334000 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
09:34:15.336000 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
09:34:15.382000 [0-0] * schannel: disabled automatic use of client certificate
09:34:15.387000 [0-0] * ALPN: curl offers http/1.1
09:34:15.389000 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
09:34:15.391000 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
09:34:15.394000 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
09:34:15.414000 [0-0] * Recv failure: Connection was reset
09:34:15.417000 [0-0] * schannel: failed to receive handshake, SSL/TLS connection failed
09:34:15.420000 [0-0] * [HTTPS-CONNECT] connect, all attempts failed
09:34:15.422000 [0-0] * [HTTPS-CONNECT] connect -> 35, done=0
09:34:15.424000 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 35, done=0
09:34:15.426000 [0-0] * [HTTPS-CONNECT] Curl_conn_connect(), filter returned 35
09:34:15.428000 [0-0] * closing connection #0
curl: (35) Recv failure: Connection was reset
```

And here is also the output from `openssl s_client -connect secure.REDACTED.com:443`

```
Connecting to REDACTED.166
CONNECTED(00000278)
write:errno=10054
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 339 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
```

My gateway is defined with the following config

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: internal-gw
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
gatewayClassName: {{ .Values.gateway.names.private }}
logging:
level:
default: debug

listeners:
- name: http-tools-argo
port: 80
protocol: HTTP
hostname: argocd.tools.REDACTED.com
- name: http-root
port: 80
protocol: HTTP
hostname: REDACTED.com
- name: https-secure
port: 443
protocol: HTTPS
hostname: secure.REDACTED.com
tls:
mode: Terminate
certificateRefs:
- kind: Secret
group: ""
name: cert-secure-REDACTED-dev
allowedRoutes:
namespaces:
from: All
```

This is the httproute resource
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: argocd-route-root
spec:
parentRefs:
- name: internal-gw
hostnames:
- "secure.REDACTED.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: argo-argocd-server
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
```

I've tested also already with the `sectionName` in the parentRef section, but also no success with it.

Gatewayclass
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: {{ .Values.gateway.names.private }}
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: {{ .Values.gateway.names.private }}
namespace: {{ .Release.Namespace }}
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: {{ .Values.gateway.names.private }}
namespace: {{ .Release.Namespace }}
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 2
envoyService:
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
service.beta.kubernetes.io/aws-load-balancer-name: "private-nlb"
service.beta.kubernetes.io/aws-load-balancer-attributes: "dns_record.client_routing_policy=availability_zone_affinity"
```

The secret
```yaml
Name: cert-secure-REDACTED-dev
Namespace: management
Labels: controller.cert-manager.io/fao=true
Annotations: cert-manager.io/alt-names: secure.REDACTED.com
cert-manager.io/certificate-name: cert-secure-REDACTED-dev
cert-manager.io/common-name: secure.REDACTED.com
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: letsencrypt-staging
cert-manager.io/uri-sans:

Type: kubernetes.io/tls

Data
====
tls.crt:
tls.key:
```

The certificate
```yaml
Name: cert-secure-REDACTED-dev
Namespace: management
Labels: app.kubernetes.io/managed-by=Helm
Annotations:
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2025-08-04T07:28:57Z
Generation: 1
Owner References:
API Version: gateway.networking.k8s.io/v1
Block Owner Deletion: true
Controller: true
Kind: Gateway
Name: internal-gw
UID: REDACTED
Resource Version: 1288691
UID: REDACTED
Spec:
Dns Names:
secure.REDACTED.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: cert-secure-REDACTED-dev
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2025-08-04T07:29:02Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2025-11-02T06:30:28Z
Not Before: 2025-08-04T06:30:29Z
Renewal Time: 2025-10-03T06:30:28Z
Revision: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 50m cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 50m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "cert-secure-REDACTED-dev-r6s25"
Normal Requested 50m cert-manager-certificates-request-manager Created new CertificateRequest resource "cert-secure-REDACTED-dev-1"
Normal Issuing 50m cert-manager-certificates-issuing The certificate has been successfully issued
```

The NLB is created, the SG allows traffic on ports 80 and 443. Also the health checks for the targetGroup is green. Cert-manager picked up the config and also created the certificate and the secret. Everything is located in the same namespace

Does anyone have an idea what is the problem?

Thanks & BR
Daniel