False consul synced check message from consul agent after VM reboot

We are using Consul v1.14.3.

We could see false health check messages from consul agent service check after VM reboot.

Our all agents are registering their service using below json template:
{
“name”: “Service Name”,
“tags”: [“Tag”],
“address”: “container IP”,
“checks”: [
{
“args”: [“sh”, “example.sh”],
“interval”: “5s”,
“timeout”: “55s”
}

#### Reproduction Steps

The issue is able to reproduce after VM reboot.

The expectation is after VM reboot until the check script “example.sh” pass the fqdn will not resolve the that particular container IP. But We could see every time that after VM reboot before agent application service comes up, from consul log we could see the service synced message. During the same time one of our client connected to this agent and do consul request over the DNS got failed since the Agent service itself dint came up completely.

Below are the agent logs from consul:

After VM reboot consul agent on application started at < 2023-08-11T11:33:21 >
2023-08-11T11:33:21.500+0530 [INFO] agent: Consul agent running!
2023-08-11T11:33:39.100+0530 [INFO] agent.client.serf.lan: serf: EventMemberJoin: Agent1 IP
2023-08-11T11:33:40.187+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:33:53.721+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:34:14.054+0530 [INFO] agent: Synced check: check=service:Service Name << **Health script has not passed at this time stamp** >>
2023-08-11T11:34:24.181+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:34:29.873+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:34:35.641+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:36:44.867+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:36:47.065+0530 [INFO] agent: Synced check: check=service:Service Name << **Health script has not passed at this time stamp** >>
2023-08-11T11:37:07.916+0530 [WARN] agent: Check is now critical: check=service:Service Name
2023-08-11T11:37:14.048+0530 [INFO] agent: Synced check: check=service:Service Name << **Application check script passed only at this time stamp** >>

From the above logs we can see there are two false positive messages from consul. When application quering agent fqdn and it resolved to that this node they got 500 response code since the application itself not started.

Below is the application exception during this time stamp:
Aug 11 11:36:56.699925 10.7.1.29 [AMQP Connection 10.7.1.16:5671, INFO, KnRMQConnListener, shutdownCompleted(ShutdownSignalException), Received shutdown completed event. ShutdownSignalException is , com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method(reply-code=541, reply-text=INTERNAL_ERROR - Cannot declare a queue ‘queue ‘’ in vhost '’’ on node ‘Agent Node’: {‘EXIT’,{aborted,{no_exists,[rabbit_vhost,<<“*****”>>]}}}, class-id=50, method-id=10)]…

**Note:**
The same issue observed on all agent nodes who registered their services with check script.
There is no consul leader fluctuation during this time stamp.

Issue is already reported here < https://discuss.hashicorp.com/t/false-consul-check-message-for-service-check-after-vm-reboot/57460> But no response.

Could you please help us to understand this issue and let me know if any additional information is required?

Start with the service-registration JSON and example.sh, then reproduce the VM reboot sequence while comparing Consul agent logs with application startup. Done would require identifying why a service is reported as synced before its check passes and documenting or testing the expected DNS behavior; no source files or tests are named.

Written by the indexing model from the issue text.