Kubernetes Performance bottleneck

*Description*:>

Describe the issue.
I have installed latest version of envoy gateway 1.4.0 on GKE version 1.32 with dataplanev2 enabled.
To install envoy gateway, we choose a node of type c4a-highcpu-32,
Modified several parameters on node level:
net.netfilter.nf_conntrack_tcp_timeout_established=54000
net.core.netdev_max_backlog=10000
net.netfilter.nf_conntrack_max=2097152
net.core.netdev_max_backlog=65535
net.ipv4.tcp_max_syn_backlog=65535
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 45000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_synack_retries = 3
net.core.somaxconn=65535

I have modified following parameters on envoy gateway level
jsonPatches:
- {"op": "add", "path": "/overload_manager/resource_monitors/0/typed_config/max_active_downstream_connections", "value": "750000"}

Created a BackendTrafficPolicy with following parameters
timeout:
http:
requestTimeout: 2s # Wait up to 2 seconds for upstream to respond
retry:
numRetries: 0 # No retries

Also applied the jsonpatch using envoyPatchPolicy:

jsonPatches:
# 1. Circuit Breakers
- name: route_name
type: type.googleapis.com/envoy.config.cluster.v3.Cluster
operation:
op: add
path: /circuit_breakers
value:
thresholds:
- priority: DEFAULT
max_connections: 250000
max_pending_requests: 50000
max_requests: 250000
max_retries: 0

# 2. TCP Keepalive
- name: route_name
type: type.googleapis.com/envoy.config.cluster.v3.Cluster
operation:
op: add
path: /upstream_connection_options
value:
tcp_keepalive:
keepalive_time: 60

# 3. Connection Timeout
- name: route_name
type: type.googleapis.com/envoy.config.cluster.v3.Cluster
operation:
op: replace
path: /connect_timeout
value: 10s

# 4. Idle Timeout
- name: route_name
type: type.googleapis.com/envoy.config.cluster.v3.Cluster
operation:
op: add
path: /common_http_protocol_options
value:
idle_timeout: 60s

The result i am getting is like system is not able to scale linearly, it reach to upstream response 2xx to 83K, then drop down, and at same time upstream pending connection increased, but number of downstream connections increased as expected

Second question: Why application/client is making too many downstream connection that increased the load on connection tracking table as well as on established socket limit.

With another solution, not kubernetes on VM (HAproxy), we get with same compute upto 150K.

[optional *Relevant Links*:]
>Any extra documentation required to understand the issue.
Please find attached screenshots: