API Gateway LoadBalancer unhelathy target

#### Overview of the Issue

I have set up consul agent on my kubernetes cluster, running on EKS.
I am trying to set up an API Gateway behind an internal load balancer with a TCP listener, to allow accesing the services in the service mash from within the VPC.
I am able to create the gateway, however the load balancer shows an unhealthy target.

---

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create EKS cluster
2. Install consul, in my case using helm.
3. Install a API Gateway, I followed the steps [here](https://developer.hashicorp.com/consul/tutorials/get-started-kubernetes/kubernetes-gs-ingress)
4. When I try to access the service at the route provided by the load balancer - I can't.
5. Also if I look in load balancer - I see that it has a target group which points to the api-gateway pod, the target show as unhelathy - failed health check.
6. When I try to access the pod at the specified port (using netcat) - I also get connection refused.
7. When I go to the Consul UI, I am able to see my services there, including the API Gateway service which shows green.

I have been trying to get it to work for the last few days, and at some point it did, but then something changed and it didn't work again.
I have tried using both an "ip" and an "instance" type NLBs.

I am installing it on a EKS cluster, using terraform.

Here are some of the relevant configurations

The values file for the helm install
```
# Contains values that affect multiple components of the chart.
global:
# The main enabled/disabled setting.
# If true, servers, clients, Consul DNS and the Consul UI will be enabled.
enabled: true
# The prefix used for all resources created in the Helm chart.
name: consul
# The consul image version.
image: hashicorp/consul:1.14.4
# The name of the datacenter that the agents should register as.
datacenter: dc1
# Enables TLS across the cluster to verify authenticity of the Consul servers and clients.
tls:
enabled: true
# Enables ACLs across the cluster to secure access to data and APIs.
acls:
# If true, automatically manage ACL tokens and policies for all Consul components.
manageSystemACLs: true
# Configures values that configure the Consul server cluster.
server:
enabled: true
# The number of server agents to run. This determines the fault tolerance of the cluster.
replicas: 3
# Contains values that configure the Consul UI.
ui:
enabled: true
# Registers a Kubernetes Service for the Consul UI as a LoadBalancer.
service:
enabled: true
type: LoadBalancer
port:
http: 8500
https: 8501
annotations: |
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
# Configures and installs the automatic Consul Connect sidecar injector.
connectInject:
enabled: true
# Configures and installs the Consul API Gateway.
apiGateway:
enabled: true
# Image to use for the api-gateway-controller pods and gateway instances
image: "hashicorp/consul-api-gateway:0.5.1"
# Configuration settings for the default GatewayClass which consul will use.
managedGatewayClass:
serviceType: LoadBalancer
copyAnnotations:
service:
annotations: |
- service.beta.kubernetes.io/aws-load-balancer-type
- service.beta.kubernetes.io/aws-load-balancer-scheme
- service.beta.kubernetes.io/aws-load-balancer-nlb-target-type
```

The configuration for the Gateway

```
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: api-gateway
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip" # Tried instance also
spec:
gatewayClassName: consul-api-gateway
# Configures the listener that is bound to the gateway's address.
listeners:
# Defines the listener protocol (HTTP, HTTPS, or TCP)
- protocol: TCP
port: 8080
name: http
allowedRoutes:
namespaces:
from: Same
```