High failed connection rate with remote ingress & egress proxies

## Overview of the Issue

When using consul service mesh, with `consul server` running on 3 hosts, and using `consul connect proxy` to accept incoming (ingress mode) connections on **NOT** `consul server` hosts (e.g.: a Remote Host); the connections created by `consul connect proxy` (running in egress mode) will fail at an extremely predictable rate dialing the incorrect host. This adds a staggering number of network errors into the service application.

This issue appears to be caused by:

1. The [random nature](https://github.com/hashicorp/consul/blob/9f46759ba2873d03a6e6a8857203deb5d1d6177b/connect/resolver.go#L105-#L109) of `consul connect proxy` (running in egress mode) selection of a health record to dial.
2. [consul creating its own health checks without `Service.Address` information](https://gist.github.com/valarauca/ff82b36199b9a09e87e95229ee8c263d) (see the `-ttl` check, it contains no `Service.Address` field). This is the information I see returned from `v1/health/connect/${service}` which I believe is what the [`health.Connect`](https://github.com/hashicorp/consul/blob/9f46759ba2873d03a6e6a8857203deb5d1d6177b/connect/resolver.go#L96) call makes (as [`Connect`](https://github.com/hashicorp/consul/blob/9f46759ba2873d03a6e6a8857203deb5d1d6177b/api/health.go#L279-#L289) invokes [`service`](https://github.com/hashicorp/consul/blob/9f46759ba2873d03a6e6a8857203deb5d1d6177b/api/health.go#L296-#L300]).
2. [The fall back logic of `consul connect proxy`](https://github.com/hashicorp/consul/blob/9f46759ba2873d03a6e6a8857203deb5d1d6177b/connect/resolver.go#L134-#L138) dialing the end point which replied to the `v1/health/connect/${service}`. This means the `consul connect proxy` running in egress mode will dial the API end point, instead of the host service.

The end resulting being `consul connect proxy` connections will fail at a rate of :

```
failure_rate in % = (1 / (total_consul_servers + 1)) * 100
```

This means with 3 `consul server` components I see a failure rate of 25%, for **EVERY** connection which crosses the consul service mesh (between hosts where the `ingress` proxy is no colocated with a `consul server` component). This has added an extremely high error rate which has prevent us from running several legacy applications within the consul-service mesh.

#### Reproduction Steps

1. Create a cluster with 3 `consul server` hosts
2. Configure the cluster to add a `service defination` for a service running on a host REMOTE to the 3 `consul server` hosts.
3. On another host REMOTE from the `consul server` hosts, configure it (as a `consul connect proxy` ) as an `egress` end point to direct traffic to the `service defination`.
4. Configure health checks for the `service defination`
5. Check `consul operator raft list-peers` to ensure all `consul server`'s have joined the quorum.
6. Check `v1/health/state/passing` to observe everything is healthy.
7. Observe as 25% of the connections to `consul connect proxy` will attempt to dial the `consul server` host, instead of the `consul connect proxy` configured as an ingress host.

**NOTE**: Setting the `"enable_tag_override": true` within the service definition does not effect the problem.

### Consul info for both Client and Server

Omitted, as this has not occurred on a single cluster, or single client.

I can provide this information is needed, but the cluster is healthy.

### Operating system and Environment details

I have replicated this error on:

On the follow distro's:

* GCP CentOS7.X (arch AMD64)
* GCP Rehat Linux 7.X (arch amd64)
* AWS Redhat Linux 7.4 (arch amd64)
* AWS Redhat Linux 7.3 (arch amd64)
* AWS Redhat Linux 7.2 (arch amd64)

On the following topologies:

* 13 host cluster (3 consul servers, ~180 proxies, 80 services) on all the above distros
* 12 host cluster (6 consul servers (split into 2 data centers), ~144 proxies, 80 services) on all the above distros
* 9 host cluster (3 consul servers, ~80 proxies, 40 services) on all the above distros
* 5 host cluster (3 consul servers, ~40 proxies, 20 services) on all the above distros

On consul versions:

* 1.5.2
* 1.6.0
* 1.6.1
* 1.6.2
* 1.7.2

### Log Fragments

example:


Same error, Different Service:


Same error on a different service.


### Question

If there anyone to disable this consul generated `-ttl` check?

Start in connect/resolver.go, especially the health selection and fallback logic linked in the report, and trace the health.Connect path through api/health.go. Reproduce the three-server setup with remote ingress and egress proxies, then inspect v1/health/connect/${service} responses and the generated -ttl check. Done means egress connections no longer dial a Consul server endpoint incorrectly and the reported failure rate is eliminated.

Written by the indexing model from the issue text.