Client-Side Rate Limiting Issue with Consul Agents in Kubernetes Environment

### Overview of the Issue

We encountered a rate-limiting issue when scaling up many new nodes with Consul agents in our Kubernetes clusters. New agents use the Kubernetes Auth Method for login and exceed the consul server leader's default QPS and Burst settings of the Kubernetes client hitting the token review API. This issue is causing a significant bottleneck, limiting the number of new nodes that can be brought online at one time. The Consul Server is unable to complete the consul agent's logins once the Kubernetes client-go client-side rate limiting hits, and the leader gets stuck in an infinite loop of retries with increasing backoff time. The leader consul server will be trapped in this retry backoff loop until we remove the pressure, or a new leader is elected.

Resolving this issue will significantly improve the scalability and reliability of Consul services in Kubernetes environments. We would greatly appreciate assistance in addressing this problem.

---

### Reproduction Steps

1. Create a cluster with 3 server nodes
2. Create >300 consul agents at once using Kubernetes Auth Method
3. Observe reaching the client-side rate limit log on consul server

### Expected Behavior

The Consul agents should scale without hitting Kubernetes client-side rate limits, allowing for a smooth and efficient scaling process.

### Actual Behavior

The leader consul server is reaching the default rate limits within the Kubernetes client-go package, preventing new nodes and their consul agents from coming online as expected. The leader gets stuck in an infinite loop of retries with increasing backoff time.

### Workaround

The workaround for this problem is to manually delete or reduce the new nodes and consul agents to relieve pressure on the consul server. Once stuck in the infinite loop retrying with a long backoff, we must do a consul leave on the leader to reset the rate limit.

### Proposed Resolution

A proposed resolution for this problem is exposing the QPS and Burst parameters for the Kubernetes auth method validator client as configurable variables for the server pods defaulted to the upstream defaults. We have tested increasing these parameters and confirmed that it eliminates the client-side rate-limiting issue and does not overwhelm the control plane. By modifying these limits in the code and rebuilding the Consul binary, we were able to eliminate the client-side rate limiting issue. Our testing with configuring QPS and Burst parameters of the validator client confirms that rate limiting no longer occurs for the consul server with sufficient QPS and Burst.

The proposed resolution does not remove the client-side rate limiting altogether but allows cluster operators to configure the QPS and Burst values according to their cluster size and performance. By default, the Kubernetes Auth Method validator client should still use the same settings as the default Kubernetes client-go library, so there will be no change in behavior for existing clusters. Cluster operators can adjust these settings based on their own testing and monitoring results and ensure that their control plane can handle the increased load without affecting other Kubernetes API clients.

### Impact

The current rate limiting issue has the potential to impact service deployment and scaling within large Kubernetes clusters. Addressing this bug is crucial for maintaining operational efficiency and reliability for clusters that support burst workloads.

### Consul info for both Client and Server

Client info

```hcl
consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = enabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 16
goroutines = 53
max_procs = 16
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 193
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 655959
members = 6
query_queue = 0
query_time = 1
```

```json
{
"enable_central_service_config": true,
"check_update_interval": "0s",
"disable_host_node_id": false,
"telemetry": {
"prometheus_retention_time": "30s",
"disable_hostname": true,
},
"limits": {
"http_max_conns_per_client": -1,
"rpc_max_conns_per_client": -1,
},
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"tokens": {
"agent": "REDACTED",
},
},
}
```

Server info

```hcl
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 6
leader = true
leader_addr = REDACTED:8300
server = true
raft:
applied_index = 9108175
commit_index = 9108175
fsm_pending = 0
last_contact = 0
last_log_index = 9108175
last_log_term = 756
last_snapshot_index = 9097562
last_snapshot_term = 756
latest_configuration = [REDACTED]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 756
runtime:
arch = amd64
cpu_count = 16
goroutines = 483
max_procs = 16
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 193
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 655955
members = 6
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 8662
members = 18
query_queue = 0
query_time = 1
```

```json
{
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_replication": true,
"enable_token_persistence": true
},
"enable_central_service_config": true,
"leave_on_terminate": true,
"telemetry": {
"prometheus_retention_time": "30s",
"disable_hostname": true
},
"limits": {
"http_max_conns_per_client": -1,
"rpc_max_conns_per_client": -1
},
"primary_datacenter": "REDACTED",
"primary_gateways": [],
"connect": {
"enable_mesh_gateway_wan_federation": true
},
"bind_addr": "0.0.0.0",
"bootstrap_expect": 3,
"client_addr": "0.0.0.0",
"connect": {
"enabled": true
},
"datacenter": "REDACTED",
"data_dir": "/consul/data",
"domain": "consul",
"ports": {
"serf_lan": 8301,
"grpc": 8503
},
"recursors": [],
"retry_join": ["REDACTED-consul-server.consul-b.svc:8301"],
"server": true,
"telemetry": {
"prometheus_retention_time": "1m"
},
"ca_file": "/consul/tls/ca/tls.crt",
"cert_file": "/consul/tls/server/tls.crt",
"key_file": "/consul/tls/server/tls.key",
"verify_incoming_rpc": true,
"verify_outgoing": true,
"verify_server_hostname": true,
"ports": {
"http": -1,
"https": 8501
},
"ui_config": {
"enabled": true
}
}
```

### Operating system and Environment details

* Consul Version: 1.13.2
* Consul-k8s Chart Version: 0.49.0
* Kubernetes Client-go Version: 1.18.2
* Kubernetes Cluster Version: 1.25.x

### Log Fragments

```log
Throttling request took 16m47.985873452s, request: POST:https://0.0.0.0/apis/authentication.k8s.io/v1/tokenreviews
```

Full logs Here: https://gist.github.com/brainshell/660ede2a1830c7d01083a8972ce7318d

### Additional Context

* Identified locations in the source code with default rate limits.

#### Source Code

Validator Instantiation

https://github.com/hashicorp/consul/blob/0e046bbb34f12b4a38c6c2dc6497426761f0b02f/agent/consul/authmethod/kubeauth/k8s.go#L103-L113

Kube Client Config Defaults

```go
// Config holds the common attributes that can be passed to a Kubernetes client on
// initialization.
type Config struct {
...

// QPS indicates the maximum QPS to the master from this client.
// If it's zero, the created RESTClient will use DefaultQPS: 5
QPS float32

// Maximum burst for throttle.
// If it's zero, the created RESTClient will use DefaultBurst: 10.
Burst int

...
}
```

https://github.com/kubernetes/client-go/blob/6b7c68377979c821b73d98d1bd4c5a466034f491/rest/config.go#L116-L120