Consul HealthCheck timeout

#### Overview of the Issue

consul -v
Consul v1.9.5
Revision 3c1c22679
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Conhar python script runs every 15s and has a timeout value of 90s.
At leat once a day, we the error below showing conhar is not completed within expected time:

**2023-12-08T02:31:01.050Z [WARN] agent: Timed out running check: check=conhar timeout=1m30s
2023-12-08T02:31:01.060Z [INFO] agent: Synced check: check=conhar
2023-12-08T02:31:29.455Z [INFO] agent.client: client starting leave
2023-12-08T02:31:29.958Z [INFO] agent.client.serf.lan: serf: EventMemberLeave: ncm-0 10.10.0.225
2023-12-08T02:31:33.758Z [INFO] agent: Requesting shutdown
2023-12-08T02:31:33.758Z [INFO] agent.client: shutting down client
2023-12-08T02:31:33.775Z [INFO] agent: consul client down
2023-12-08T02:31:33.775Z [INFO] agent: shutdown complete**

However, this script does not take more than a few seconds to complete.
I inserted instructions below to have an idea how long it takes:
``
start_time = time.time()
start_date_time = time.strftime("%Y-%m-%d %H:%M:%S")
{...}
end_time = time.time()
execution_time = end_time - start_time
with open("/ericsson/enm/dumps/ncm/conhar_execution_time.txt", "a") as log_file:
log_file.write(f"Start Date Time: {start_date_time}\n")
log_file.write(f"Script execution time: {execution_time:.5f} seconds\n")
log_file.write("\n")
``


Here is the thing... when we have Timed out running check: check=conhar timeout=1m30s, there is no evidence that the script took more than 90s to complete.
Also, we do not see performance issues that could affect it.

#### Reproduction Steps

Well, honestly I could not find a way to reproduce it.

### Consul info for both Client and Server

Server
consul info
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 3c1c2267
version = 1.9.5
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.10.0.196:8300
server = true
raft:
applied_index = 176131
commit_index = 176131
fsm_pending = 0
last_contact = 0
last_log_index = 176131
last_log_term = 2
last_snapshot_index = 163931
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:604fabad-2a98-d52b-af4e-a59cde322db2 Address:10.10.0.197:8300} {Suffrage:Voter ID:9f243259-94ab-b32e-24c7-061f5adce9d4 Address:10.10.0.198:8300} {Suffrage:Voter ID:01b36e27-a81d-aed3-1ab9-2faf7a37a951 Address:10.10.0.196:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 1
goroutines = 742
max_procs = 1
os = linux
version = go1.15.8
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 525
members = 180
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6
members = 3
query_queue = 0
query_time = 1

Client info

```
Output from client 'consul info' command here
```

```
Client agent HCL config
```

Server info

```
Output from server 'consul info' command here
```
AGENT

agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 5
build:
prerelease =
revision = 3c1c2267
version = 1.9.5
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 59
max_procs = 4
os = linux
version = go1.15.8
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 1
member_time = 504
members = 180
query_queue = 0
query_time = 1
```
Server agent HCL config
```

### Operating system and Environment details

NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp4"
DOCUMENTATION_URL="[https://documentation.suse.com/"](https://documentation.suse.com/%22)

Ram: 12G
CPU: AMD EPYC 7713P 64-Core Processor

### Log Fragments