proxies are reported alive, even if they are not.

#### Overview of the Issue

Attempting to centralize a configuration for `consul agent` to cut down on clutter, and I ran into some difficulty.

Namely a `consul connect proxy` which fails to get a SPIFFEE cert, and never opens its local ports, is reported as alive & healthy.

#### Reproduction Steps

Create a RAFT clusters of 5 servers. I used the _same_ configuration for all 3. It is as follows:

Server configuration

```json
{
"addresses": {
"http": "127.0.0.1",
"https": "0.0.0.0"
},
"bootstrap": false,
"bootstrap_expect": 3,
"ca_file": "/opt/consul/certs/ca_cert.pem",
"cert_file": "/opt/consul/certs/local_cert.pem",
"data_dir": "/opt/consul/data",
"discard_check_output": null,
"discovery_max_stale": null,
"enable_script_checks": false,
"enable_local_script_checks": false,
"encrypt": "hLRmojjbMVq4hpIHrBRkCw==",
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"key_file": "/opt/consul/certs/local_key.pem",
"leave_on_terminate": true,
"log_level": "TRACE",
"log_file": "/opt/consul/consul.log",
"log_rotate_bytes": 1048576,
"pid_file": "/opt/consul/apigee-consul.pid",
"ports": {
"http": 8500,
"https": 8503,
"grpc": -1
},
"rejoin_after_leave": true,
"retry_interval": "5s",
"retry_join": [
"10.126.15.192",
"10.126.15.193",
"10.126.15.194",
"10.126.15.195",
"10.126.15.196"
],
"retry_max": 200,
"server": true,
"services": [
{
"kind": "connect-proxy",
"id": "zookeeper-2181-10-126-15-192",
"name": "zookeeper-2181-10-126-15-192",
"port": 10000,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-2181-10-126-15-192",
"destination_service_id": "zookeeper-2181-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 2181
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-2888-10-126-15-192",
"name": "zookeeper-2888-10-126-15-192",
"port": 10001,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-2888-10-126-15-192",
"destination_service_id": "zookeeper-2888-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 2888
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-3888-10-126-15-192",
"name": "zookeeper-3888-10-126-15-192",
"port": 10002,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-3888-10-126-15-192",
"destination_service_id": "zookeeper-3888-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 3888
}
},
{
"kind": "connect-proxy",
"id": "cassandra-9042-10-126-15-192",
"name": "cassandra-9042-10-126-15-192",
"port": 10003,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "cassandra-9042-10-126-15-192",
"destination_service_id": "cassandra-9042-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 9042
}
},
{
"kind": "connect-proxy",
"id": "cassandra-9160-10-126-15-192",
"name": "cassandra-9160-10-126-15-192",
"port": 10004,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "cassandra-9160-10-126-15-192",
"destination_service_id": "cassandra-9160-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 9160
}
},
{
"kind": "connect-proxy",
"id": "message-router-4527-10-126-15-192",
"name": "message-router-4527-10-126-15-192",
"port": 10005,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "message-router-4527-10-126-15-192",
"destination_service_id": "message-router-4527-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 4527
}
},
{
"kind": "connect-proxy",
"id": "message-processor-4528-10-126-15-192",
"name": "message-processor-4528-10-126-15-192",
"port": 10006,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "message-processor-4528-10-126-15-192",
"destination_service_id": "message-processor-4528-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 4528
}
},
{
"kind": "connect-proxy",
"id": "message-processor-8082-10-126-15-192",
"name": "message-processor-8082-10-126-15-192",
"port": 10007,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "message-processor-8082-10-126-15-192",
"destination_service_id": "message-processor-8082-10-126-15-192",
"local_service_address": "127.0.0.1",
"local_service_port": 8082
}
},
{
"kind": "connect-proxy",
"id": "egress-10-126-15-192",
"name": "egress-10-126-15-192",
"port": 10032,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "egress-10-126-15-192",
"destination_service_id": "egress-10-126-15-192",
"upstreams": [
{
"destination_name": "qpid-4529-10-126-15-193",
"local_bind_port": 10008
},
{
"destination_name": "qpid-5672-10-126-15-193",
"local_bind_port": 10009
},
{
"destination_name": "qpid-4529-10-126-15-194",
"local_bind_port": 10013
},
{
"destination_name": "qpid-5672-10-126-15-194",
"local_bind_port": 10014
},
{
"destination_name": "zookeeper-2181-10-126-15-195",
"local_bind_port": 10018
},
{
"destination_name": "zookeeper-2888-10-126-15-195",
"local_bind_port": 10019
},
{
"destination_name": "zookeeper-3888-10-126-15-195",
"local_bind_port": 10020
},
{
"destination_name": "cassandra-9042-10-126-15-195",
"local_bind_port": 10021
},
{
"destination_name": "cassandra-9160-10-126-15-195",
"local_bind_port": 10022
},
{
"destination_name": "zookeeper-2181-10-126-15-196",
"local_bind_port": 10024
},
{
"destination_name": "zookeeper-2888-10-126-15-196",
"local_bind_port": 10025
},
{
"destination_name": "zookeeper-3888-10-126-15-196",
"local_bind_port": 10026
},
{
"destination_name": "cassandra-9042-10-126-15-196",
"local_bind_port": 10027
},
{
"destination_name": "cassandra-9160-10-126-15-196",
"local_bind_port": 10028
},
{
"destination_name": "message-processor-4528-10-126-15-196",
"local_bind_port": 10030
},
{
"destination_name": "message-processor-8082-10-126-15-196",
"local_bind_port": 10031
}
]
}
},
{
"kind": "connect-proxy",
"id": "qpid-4529-10-126-15-193",
"name": "qpid-4529-10-126-15-193",
"port": 10008,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "qpid-4529-10-126-15-193",
"destination_service_id": "qpid-4529-10-126-15-193",
"local_service_address": "127.0.0.1",
"local_service_port": 4529
}
},
{
"kind": "connect-proxy",
"id": "qpid-5672-10-126-15-193",
"name": "qpid-5672-10-126-15-193",
"port": 10009,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "qpid-5672-10-126-15-193",
"destination_service_id": "qpid-5672-10-126-15-193",
"local_service_address": "127.0.0.1",
"local_service_port": 5672
}
},
{
"kind": "connect-proxy",
"id": "postgresql-4530-10-126-15-193",
"name": "postgresql-4530-10-126-15-193",
"port": 10010,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "postgresql-4530-10-126-15-193",
"destination_service_id": "postgresql-4530-10-126-15-193",
"local_service_address": "127.0.0.1",
"local_service_port": 4530
}
},
{
"kind": "connect-proxy",
"id": "postgresql-5432-10-126-15-193",
"name": "postgresql-5432-10-126-15-193",
"port": 10011,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "postgresql-5432-10-126-15-193",
"destination_service_id": "postgresql-5432-10-126-15-193",
"local_service_address": "127.0.0.1",
"local_service_port": 5432
}
},
{
"kind": "connect-proxy",
"id": "postgresql-8084-10-126-15-193",
"name": "postgresql-8084-10-126-15-193",
"port": 10012,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "postgresql-8084-10-126-15-193",
"destination_service_id": "postgresql-8084-10-126-15-193",
"local_service_address": "127.0.0.1",
"local_service_port": 8084
}
},
{
"kind": "connect-proxy",
"id": "egress-10-126-15-193",
"name": "egress-10-126-15-193",
"port": 10033,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "egress-10-126-15-193",
"destination_service_id": "egress-10-126-15-193",
"upstreams": [
{
"destination_name": "zookeeper-2181-10-126-15-192",
"local_bind_port": 10000
},
{
"destination_name": "zookeeper-2888-10-126-15-192",
"local_bind_port": 10001
},
{
"destination_name": "zookeeper-3888-10-126-15-192",
"local_bind_port": 10002
},
{
"destination_name": "cassandra-9042-10-126-15-192",
"local_bind_port": 10003
},
{
"destination_name": "cassandra-9160-10-126-15-192",
"local_bind_port": 10004
},
{
"destination_name": "qpid-4529-10-126-15-194",
"local_bind_port": 10013
},
{
"destination_name": "qpid-5672-10-126-15-194",
"local_bind_port": 10014
},
{
"destination_name": "postgresql-4530-10-126-15-194",
"local_bind_port": 10015
},
{
"destination_name": "postgresql-5432-10-126-15-194",
"local_bind_port": 10016
},
{
"destination_name": "postgresql-8084-10-126-15-194",
"local_bind_port": 10017
},
{
"destination_name": "zookeeper-2181-10-126-15-195",
"local_bind_port": 10018
},
{
"destination_name": "zookeeper-2888-10-126-15-195",
"local_bind_port": 10019
},
{
"destination_name": "zookeeper-3888-10-126-15-195",
"local_bind_port": 10020
},
{
"destination_name": "cassandra-9042-10-126-15-195",
"local_bind_port": 10021
},
{
"destination_name": "cassandra-9160-10-126-15-195",
"local_bind_port": 10022
},
{
"destination_name": "zookeeper-2181-10-126-15-196",
"local_bind_port": 10024
},
{
"destination_name": "zookeeper-2888-10-126-15-196",
"local_bind_port": 10025
},
{
"destination_name": "zookeeper-3888-10-126-15-196",
"local_bind_port": 10026
},
{
"destination_name": "cassandra-9042-10-126-15-196",
"local_bind_port": 10027
},
{
"destination_name": "cassandra-9160-10-126-15-196",
"local_bind_port": 10028
}
]
}
},
{
"kind": "connect-proxy",
"id": "qpid-4529-10-126-15-194",
"name": "qpid-4529-10-126-15-194",
"port": 10013,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "qpid-4529-10-126-15-194",
"destination_service_id": "qpid-4529-10-126-15-194",
"local_service_address": "127.0.0.1",
"local_service_port": 4529
}
},
{
"kind": "connect-proxy",
"id": "qpid-5672-10-126-15-194",
"name": "qpid-5672-10-126-15-194",
"port": 10014,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "qpid-5672-10-126-15-194",
"destination_service_id": "qpid-5672-10-126-15-194",
"local_service_address": "127.0.0.1",
"local_service_port": 5672
}
},
{
"kind": "connect-proxy",
"id": "postgresql-4530-10-126-15-194",
"name": "postgresql-4530-10-126-15-194",
"port": 10015,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "postgresql-4530-10-126-15-194",
"destination_service_id": "postgresql-4530-10-126-15-194",
"local_service_address": "127.0.0.1",
"local_service_port": 4530
}
},
{
"kind": "connect-proxy",
"id": "postgresql-5432-10-126-15-194",
"name": "postgresql-5432-10-126-15-194",
"port": 10016,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "postgresql-5432-10-126-15-194",
"destination_service_id": "postgresql-5432-10-126-15-194",
"local_service_address": "127.0.0.1",
"local_service_port": 5432
}
},
{
"kind": "connect-proxy",
"id": "postgresql-8084-10-126-15-194",
"name": "postgresql-8084-10-126-15-194",
"port": 10017,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "postgresql-8084-10-126-15-194",
"destination_service_id": "postgresql-8084-10-126-15-194",
"local_service_address": "127.0.0.1",
"local_service_port": 8084
}
},
{
"kind": "connect-proxy",
"id": "egress-10-126-15-194",
"name": "egress-10-126-15-194",
"port": 10034,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "egress-10-126-15-194",
"destination_service_id": "egress-10-126-15-194",
"upstreams": [
{
"destination_name": "zookeeper-2181-10-126-15-192",
"local_bind_port": 10000
},
{
"destination_name": "zookeeper-2888-10-126-15-192",
"local_bind_port": 10001
},
{
"destination_name": "zookeeper-3888-10-126-15-192",
"local_bind_port": 10002
},
{
"destination_name": "cassandra-9042-10-126-15-192",
"local_bind_port": 10003
},
{
"destination_name": "cassandra-9160-10-126-15-192",
"local_bind_port": 10004
},
{
"destination_name": "qpid-4529-10-126-15-193",
"local_bind_port": 10008
},
{
"destination_name": "qpid-5672-10-126-15-193",
"local_bind_port": 10009
},
{
"destination_name": "postgresql-4530-10-126-15-193",
"local_bind_port": 10010
},
{
"destination_name": "postgresql-5432-10-126-15-193",
"local_bind_port": 10011
},
{
"destination_name": "postgresql-8084-10-126-15-193",
"local_bind_port": 10012
},
{
"destination_name": "zookeeper-2181-10-126-15-195",
"local_bind_port": 10018
},
{
"destination_name": "zookeeper-2888-10-126-15-195",
"local_bind_port": 10019
},
{
"destination_name": "zookeeper-3888-10-126-15-195",
"local_bind_port": 10020
},
{
"destination_name": "cassandra-9042-10-126-15-195",
"local_bind_port": 10021
},
{
"destination_name": "cassandra-9160-10-126-15-195",
"local_bind_port": 10022
},
{
"destination_name": "zookeeper-2181-10-126-15-196",
"local_bind_port": 10024
},
{
"destination_name": "zookeeper-2888-10-126-15-196",
"local_bind_port": 10025
},
{
"destination_name": "zookeeper-3888-10-126-15-196",
"local_bind_port": 10026
},
{
"destination_name": "cassandra-9042-10-126-15-196",
"local_bind_port": 10027
},
{
"destination_name": "cassandra-9160-10-126-15-196",
"local_bind_port": 10028
}
]
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-2181-10-126-15-195",
"name": "zookeeper-2181-10-126-15-195",
"port": 10018,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-2181-10-126-15-195",
"destination_service_id": "zookeeper-2181-10-126-15-195",
"local_service_address": "127.0.0.1",
"local_service_port": 2181
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-2888-10-126-15-195",
"name": "zookeeper-2888-10-126-15-195",
"port": 10019,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-2888-10-126-15-195",
"destination_service_id": "zookeeper-2888-10-126-15-195",
"local_service_address": "127.0.0.1",
"local_service_port": 2888
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-3888-10-126-15-195",
"name": "zookeeper-3888-10-126-15-195",
"port": 10020,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-3888-10-126-15-195",
"destination_service_id": "zookeeper-3888-10-126-15-195",
"local_service_address": "127.0.0.1",
"local_service_port": 3888
}
},
{
"kind": "connect-proxy",
"id": "cassandra-9042-10-126-15-195",
"name": "cassandra-9042-10-126-15-195",
"port": 10021,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "cassandra-9042-10-126-15-195",
"destination_service_id": "cassandra-9042-10-126-15-195",
"local_service_address": "127.0.0.1",
"local_service_port": 9042
}
},
{
"kind": "connect-proxy",
"id": "cassandra-9160-10-126-15-195",
"name": "cassandra-9160-10-126-15-195",
"port": 10022,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "cassandra-9160-10-126-15-195",
"destination_service_id": "cassandra-9160-10-126-15-195",
"local_service_address": "127.0.0.1",
"local_service_port": 9160
}
},
{
"kind": "connect-proxy",
"id": "ldap-10389-10-126-15-195",
"name": "ldap-10389-10-126-15-195",
"port": 10023,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "ldap-10389-10-126-15-195",
"destination_service_id": "ldap-10389-10-126-15-195",
"local_service_address": "127.0.0.1",
"local_service_port": 10389
}
},
{
"kind": "connect-proxy",
"id": "egress-10-126-15-195",
"name": "egress-10-126-15-195",
"port": 10035,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "egress-10-126-15-195",
"destination_service_id": "egress-10-126-15-195",
"upstreams": [
{
"destination_name": "zookeeper-2181-10-126-15-192",
"local_bind_port": 10000
},
{
"destination_name": "zookeeper-2888-10-126-15-192",
"local_bind_port": 10001
},
{
"destination_name": "zookeeper-3888-10-126-15-192",
"local_bind_port": 10002
},
{
"destination_name": "cassandra-9042-10-126-15-192",
"local_bind_port": 10003
},
{
"destination_name": "cassandra-9160-10-126-15-192",
"local_bind_port": 10004
},
{
"destination_name": "message-router-4527-10-126-15-192",
"local_bind_port": 10005
},
{
"destination_name": "message-processor-4528-10-126-15-192",
"local_bind_port": 10006
},
{
"destination_name": "message-processor-8082-10-126-15-192",
"local_bind_port": 10007
},
{
"destination_name": "qpid-4529-10-126-15-193",
"local_bind_port": 10008
},
{
"destination_name": "qpid-5672-10-126-15-193",
"local_bind_port": 10009
},
{
"destination_name": "postgresql-4530-10-126-15-193",
"local_bind_port": 10010
},
{
"destination_name": "postgresql-5432-10-126-15-193",
"local_bind_port": 10011
},
{
"destination_name": "postgresql-8084-10-126-15-193",
"local_bind_port": 10012
},
{
"destination_name": "qpid-4529-10-126-15-194",
"local_bind_port": 10013
},
{
"destination_name": "qpid-5672-10-126-15-194",
"local_bind_port": 10014
},
{
"destination_name": "postgresql-4530-10-126-15-194",
"local_bind_port": 10015
},
{
"destination_name": "postgresql-5432-10-126-15-194",
"local_bind_port": 10016
},
{
"destination_name": "postgresql-8084-10-126-15-194",
"local_bind_port": 10017
},
{
"destination_name": "zookeeper-2181-10-126-15-196",
"local_bind_port": 10024
},
{
"destination_name": "zookeeper-2888-10-126-15-196",
"local_bind_port": 10025
},
{
"destination_name": "zookeeper-3888-10-126-15-196",
"local_bind_port": 10026
},
{
"destination_name": "cassandra-9042-10-126-15-196",
"local_bind_port": 10027
},
{
"destination_name": "cassandra-9160-10-126-15-196",
"local_bind_port": 10028
},
{
"destination_name": "message-router-4527-10-126-15-196",
"local_bind_port": 10029
},
{
"destination_name": "message-processor-4528-10-126-15-196",
"local_bind_port": 10030
},
{
"destination_name": "message-processor-8082-10-126-15-196",
"local_bind_port": 10031
}
]
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-2181-10-126-15-196",
"name": "zookeeper-2181-10-126-15-196",
"port": 10024,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-2181-10-126-15-196",
"destination_service_id": "zookeeper-2181-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 2181
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-2888-10-126-15-196",
"name": "zookeeper-2888-10-126-15-196",
"port": 10025,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-2888-10-126-15-196",
"destination_service_id": "zookeeper-2888-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 2888
}
},
{
"kind": "connect-proxy",
"id": "zookeeper-3888-10-126-15-196",
"name": "zookeeper-3888-10-126-15-196",
"port": 10026,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "zookeeper-3888-10-126-15-196",
"destination_service_id": "zookeeper-3888-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 3888
}
},
{
"kind": "connect-proxy",
"id": "cassandra-9042-10-126-15-196",
"name": "cassandra-9042-10-126-15-196",
"port": 10027,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "cassandra-9042-10-126-15-196",
"destination_service_id": "cassandra-9042-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 9042
}
},
{
"kind": "connect-proxy",
"id": "cassandra-9160-10-126-15-196",
"name": "cassandra-9160-10-126-15-196",
"port": 10028,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "cassandra-9160-10-126-15-196",
"destination_service_id": "cassandra-9160-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 9160
}
},
{
"kind": "connect-proxy",
"id": "message-router-4527-10-126-15-196",
"name": "message-router-4527-10-126-15-196",
"port": 10029,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "message-router-4527-10-126-15-196",
"destination_service_id": "message-router-4527-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 4527
}
},
{
"kind": "connect-proxy",
"id": "message-processor-4528-10-126-15-196",
"name": "message-processor-4528-10-126-15-196",
"port": 10030,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "message-processor-4528-10-126-15-196",
"destination_service_id": "message-processor-4528-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 4528
}
},
{
"kind": "connect-proxy",
"id": "message-processor-8082-10-126-15-196",
"name": "message-processor-8082-10-126-15-196",
"port": 10031,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "message-processor-8082-10-126-15-196",
"destination_service_id": "message-processor-8082-10-126-15-196",
"local_service_address": "127.0.0.1",
"local_service_port": 8082
}
},
{
"kind": "connect-proxy",
"id": "egress-10-126-15-196",
"name": "egress-10-126-15-196",
"port": 10036,
"enable_tag_override": false,
"proxy": {
"destination_service_name": "egress-10-126-15-196",
"destination_service_id": "egress-10-126-15-196",
"upstreams": [
{
"destination_name": "zookeeper-2181-10-126-15-192",
"local_bind_port": 10000
},
{
"destination_name": "zookeeper-2888-10-126-15-192",
"local_bind_port": 10001
},
{
"destination_name": "zookeeper-3888-10-126-15-192",
"local_bind_port": 10002
},
{
"destination_name": "cassandra-9042-10-126-15-192",
"local_bind_port": 10003
},
{
"destination_name": "cassandra-9160-10-126-15-192",
"local_bind_port": 10004
},
{
"destination_name": "message-processor-4528-10-126-15-192",
"local_bind_port": 10006
},
{
"destination_name": "message-processor-8082-10-126-15-192",
"local_bind_port": 10007
},
{
"destination_name": "qpid-4529-10-126-15-193",
"local_bind_port": 10008
},
{
"destination_name": "qpid-5672-10-126-15-193",
"local_bind_port": 10009
},
{
"destination_name": "qpid-4529-10-126-15-194",
"local_bind_port": 10013
},
{
"destination_name": "qpid-5672-10-126-15-194",
"local_bind_port": 10014
},
{
"destination_name": "zookeeper-2181-10-126-15-195",
"local_bind_port": 10018
},
{
"destination_name": "zookeeper-2888-10-126-15-195",
"local_bind_port": 10019
},
{
"destination_name": "zookeeper-3888-10-126-15-195",
"local_bind_port": 10020
},
{
"destination_name": "cassandra-9042-10-126-15-195",
"local_bind_port": 10021
},
{
"destination_name": "cassandra-9160-10-126-15-195",
"local_bind_port": 10022
}
]
}
}
],
"skip_leave_on_interrupt": false,
"start_join": [
"10.126.15.192",
"10.126.15.193",
"10.126.15.194",
"10.126.15.195",
"10.126.15.196"
],
"verify_incoming": true,
"verify_incoming_https": true,
"verify_incoming_rpc": true,
"verify_outgoing": true,
"connect": {
"ca_config": {
"private_key": "-----BEGIN EC PRIVATE KEY----- lol -----END EC PRIVATE KEY-----\n",
"root_cert": "-----BEGIN CERTIFICATE----- lol -----END CERTIFICATE-----\n",
"csr_max_per_second": 100,
"csr_max_concurrent": 1000,
"leaf_cert_ttl": "3h0m0s"
},
"ca_provider": "consul"
}
}

```

I am fairly certain the configuration is erroneous, but that isn't the bug report.

I start my `consul proxy` via system.

systemd unit file

```
[Unit]
Description=local proxy for zookeeper:2181
Wants=network.target
After=consul.service

[Service]
ExecStart=/opt/consul/bin/consul connect proxy -register -service zookeeper-2181-10-126-15-196 -service-addr 127.0.0.1:2181 -listen :10024 -log-level TRACE
User=consul
Group=consul
StandardOutput=journal
StandardError=journal
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target
```

After starting I'll perform the following verification steps

consul operator raft list-peers

```sh
consul operator raft list-peers
Node ID Address State Voter RaftProtocol
prc-test-2-871 8d8c7c78-9f9a-efdc-d96b-3e8d8170695a 10.126.15.192:8300 follower true 3
prc-test-4-871 c5bb660b-cc82-0078-67a8-70b0c6b7d289 10.126.15.193:8300 leader true 3
prc-test-1-871 66863776-8591-3944-51bc-e96645f6b3de 10.126.15.196:8300 follower true 3
prc-test-3-871 0f6740f5-a958-f594-422e-a949a34d5401 10.126.15.194:8300 follower true 3
prc-test-0-871 9589ae27-8b95-f977-6cee-d351e776ad3c 10.126.15.195:8300 follower true 3
```

api stuff

```sh
curl http://127.0.0.1:8500/v1/catalog/service/zookeeper-2181-10-126-15-192-proxy?consistent=true
```

which returns

```json
[{"Node":{"ID":"8d8c7c78-9f9a-efdc-d96b-3e8d8170695a","Node":"prc-test-2-871","Address":"10.126.15.192","Datacenter":"dc1","TaggedAddresses":{"lan":"10.126.15.192","wan":"10.126.15.192"},"Meta":{"consul-network-segment":""},"CreateIndex":5,"ModifyIndex":9},"Service":{"Kind":"connect-proxy","ID":"zookeeper-2181-10-126-15-192-proxy","Service":"zookeeper-2181-10-126-15-192-proxy","Tags":[],"Address":"","Meta":null,"Port":10000,"Weights":{"Passing":1,"Warning":1},"EnableTagOverride":false,"ProxyDestination":"","Proxy":{"DestinationServiceName":"zookeeper-2181-10-126-15-192"},"Connect":{},"CreateIndex":225,"ModifyIndex":225},"Checks":[{"Node":"prc-test-2-871","CheckID":"serfHealth","Name":"Serf Health Status","Status":"passing","Notes":"","Output":"Agent alive and reachable","ServiceID":"","ServiceName":"","ServiceTags":[],"Definition":{},"CreateIndex":5,"ModifyIndex":5},{"Node":"prc-test-2-871","CheckID":"zookeeper-2181-10-126-15-192-proxy-ttl","Name":"proxy heartbeat","Status":"passing","Notes":"Built-in proxy will heartbeat this check.","Output":"","ServiceID":"zookeeper-2181-10-126-15-192-proxy","ServiceName":"zookeeper-2181-10-126-15-192-proxy","ServiceTags":[],"Definition":{},"CreateIndex":225,"ModifyIndex":225}]}]
```

### Details

So far my service mesh looks solid.

- [X] Raft Quorum Healthy
- [X] Proxy Alive

But when I check with `lsof` the proxy isn't listening:

lsofoutput

```
lsof -n -P -iTCP -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
master 1107 root 13u IPv4 16921 0t0 TCP 127.0.0.1:25 (LISTEN)
master 1107 root 14u IPv6 16922 0t0 TCP [::1]:25 (LISTEN)
sshd 2651 root 3u IPv4 23198 0t0 TCP *:22 (LISTEN)
sshd 2651 root 4u IPv6 23200 0t0 TCP *:22 (LISTEN)
consul 18001 consul 3u IPv6 151523 0t0 TCP *:8300 (LISTEN)
consul 18001 consul 7u IPv6 151525 0t0 TCP *:8302 (LISTEN)
consul 18001 consul 10u IPv6 151528 0t0 TCP *:8301 (LISTEN)
consul 18001 consul 15u IPv4 151531 0t0 TCP 127.0.0.1:8600 (LISTEN)
consul 18001 consul 16u IPv4 151533 0t0 TCP 127.0.0.1:8500 (LISTEN)
consul 18001 consul 17u IPv6 151534 0t0 TCP *:8503 (LISTEN)
```

Also when I look at the proxies logs

proxy logs

```
Jul 13 01:23:52 prc-test-2-871 consul[27981]: 2019/07/13 01:23:52 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:23:22 prc-test-2-871 consul[27981]: 2019/07/13 01:23:22 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:22:52 prc-test-2-871 consul[27981]: 2019/07/13 01:22:52 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:22:26 prc-test-2-871 consul[27981]: 2019/07/13 01:22:26 [ERR] consul.watch: Watch (type: connect_leaf) errored: Unexpected response code: 500 (rpc e
Jul 13 01:22:26 prc-test-2-871 consul[27981]: 2019/07/13 01:22:26 [ERR] consul.watch: Watch (type: connect_roots) errored: Unexpected response code: 500 (rpc
Jul 13 01:22:22 prc-test-2-871 consul[27981]: 2019/07/13 01:22:22 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:21:52 prc-test-2-871 consul[27981]: 2019/07/13 01:21:52 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:21:22 prc-test-2-871 consul[27981]: 2019/07/13 01:21:22 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:20:52 prc-test-2-871 consul[27981]: 2019/07/13 01:20:52 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:20:22 prc-test-2-871 consul[27981]: 2019/07/13 01:20:22 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:19:52 prc-test-2-871 consul[27981]: 2019/07/13 01:19:52 [DEBUG] proxy: service already registered, not re-registering
Jul 13 01:19:26 prc-test-2-871 consul[27981]: 2019/07/13 01:19:26 [ERR] consul.watch: Watch (type: connect_leaf) errored: Unexpected response code: 500 (rpc e
Jul 13 01:19:26 prc-test-2-871 consul[27981]: 2019/07/13 01:19:26 [ERR] consul.watch: Watch (type: connect_roots) errored: Unexpected response code: 500 (rpc
Jul 13 01:19:22 prc-test-2-871 consul[27981]: 2019/07/13 01:19:22 [DEBUG] proxy: service already registered, not re-registering
```

It appears that it hasn't received a leaf-certificate.

---

### Bug Report

If the proxy doesn't have a SPIFEE cert, and isn't listening. Why it is classified as healthy?

### Platform Details

consul - v1.5.1
amd6 Linux
centos7.X

Start by tracing the consul connect proxy health and status handling for proxies that fail to obtain a SPIFFE certificate or never open their local ports. Reproduce the reported configuration and inspect how the proxy is marked alive and healthy. Done means a failed proxy is no longer reported as alive and healthy, with coverage for this failure case.

Written by the indexing model from the issue text.