Connect upstream using prepared query doesn't seem to work

This might be the same as https://github.com/hashicorp/consul/issues/8011, but in their case they are going cross-DC in a federated setup. I can reproduce the problem using just countdash with Consul in dev mode, unless I'm just missing something.

Context: adding `upstream.destination_type` to nomad https://github.com/hashicorp/nomad/issues/13138
Consul API docs: https://www.consul.io/docs/connect/registration/service-registration#upstream-configuration-reference

```
nomad 19m54son cc-upstream-destination-type [m]
➜ nomad version
Nomad v1.3.2-dev (95c6fce4f1a5d12ce1e2fa97248c6be316cfe2db+CHANGES)
```

```
➜ consul version
Consul v1.12.1+ent
Revision cfd128ff
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
```

Nomad job file

```hcl
# cd2.nomad

job "cd2" {
datacenters = ["dc1"]

group "api" {
network {
mode = "bridge"
}

service {
name = "count-api"
port = "9001"

connect {
sidecar_service {}
}
}

task "web" {
driver = "docker"

config {
image = "hashicorpnomad/counter-api:v3"
}
}
}

group "dashboard" {
network {
mode = "bridge"

port "http" {
static = 9002
to = 9002
}
}

service {
name = "count-dashboard"
port = "9002"

connect {
sidecar_service {
proxy {
upstreams {
# destination_name = "count-api"
destination_name = "my-query"
destination_type = "prepared_query"
local_bind_port = 8080
}
}
}
}
}

task "dashboard" {
driver = "docker"

env {
COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
}

config {
image = "hashicorpnomad/counter-dashboard:v3"
}
}
}
}
```

Query

```
➜ cat query.json
{
"Name": "my-query",
"Service": {
"Service": "count-api",
"OnlyPassing": false,
"Connect": true
}
}
```

Run consul

```
➜ consul agent -dev -log-level=TRACE
```

Run nomad

```
➜ sudo nomad agent -dev-connect
```

Register query

```
➜ curl -XPOST --data @query.json localhost:8500/v1/query
{
"ID": "f06415a1-e73c-50f1-94b5-e0fc9f1ae63c"
}
```

Run nomad job

```
➜ nomad job run cd2.nomad

...
Task Group Desired Placed Healthy Unhealthy Progress Deadline
api 1 1 1 0 2022-05-26T16:11:46-05:00
dashboard 1 1 1 0 2022-05-26T16:11:46-05:00

```

At this point, `localhost:9002` should be a happy countdash, but it errors with `Counting Service is Unreachable`

Let's inspect the execution of our query

```
➜ curl -s localhost:8500/v1/query/f06415a1-e73c-50f1-94b5-e0fc9f1ae63c/execute | jq .
```

execution output

```
{
"Service": "count-api",
"Nodes": [
{
"Node": {
"ID": "35555bcd-a7a6-ae9d-3830-ea80c1a0da6b",
"Node": "x52",
"Address": "127.0.0.1",
"Datacenter": "dc1",
"Partition": "default",
"TaggedAddresses": {
"lan": "127.0.0.1",
"lan_ipv4": "127.0.0.1",
"wan": "127.0.0.1",
"wan_ipv4": "127.0.0.1"
},
"Meta": {
"consul-network-segment": ""
},
"CreateIndex": 15,
"ModifyIndex": 17
},
"Service": {
"Kind": "connect-proxy",
"ID": "_nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001-sidecar-proxy",
"Service": "count-api-sidecar-proxy",
"Tags": null,
"Address": "192.168.88.252",
"TaggedAddresses": {
"consul-virtual": {
"Address": "240.0.0.2",
"Port": 31924
},
"lan_ipv4": {
"Address": "192.168.88.252",
"Port": 31924
},
"wan_ipv4": {
"Address": "192.168.88.252",
"Port": 31924
}
},
"Meta": {
"external-source": "nomad"
},
"Port": 31924,
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Proxy": {
"DestinationServiceName": "count-api",
"DestinationServiceID": "_nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001",
"LocalServiceAddress": "127.0.0.1",
"LocalServicePort": 9001,
"Mode": "",
"Config": {
"bind_address": "0.0.0.0",
"bind_port": 31924,
"envoy_stats_tags": [
"nomad.alloc_id=0cc928cd-364f-b257-1e62-ef1b8ff25de7"
]
},
"MeshGateway": {},
"Expose": {}
},
"Connect": {},
"Partition": "default",
"Namespace": "default",
"CreateIndex": 40,
"ModifyIndex": 40
},
"Checks": [
{
"Node": "x52",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"ServiceTags": null,
"Type": "",
"Interval": "",
"Timeout": "",
"ExposedPort": 0,
"Definition": {},
"Partition": "default",
"Namespace": "default",
"CreateIndex": 15,
"ModifyIndex": 15
},
{
"Node": "x52",
"CheckID": "service:_nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001-sidecar-proxy:1",
"Name": "Connect Sidecar Aliasing _nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001",
"Status": "passing",
"Notes": "",
"Output": "No checks found.",
"ServiceID": "_nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001-sidecar-proxy",
"ServiceName": "count-api-sidecar-proxy",
"ServiceTags": null,
"Type": "alias",
"Interval": "",
"Timeout": "",
"ExposedPort": 0,
"Definition": {},
"Partition": "default",
"Namespace": "default",
"CreateIndex": 40,
"ModifyIndex": 42
},
{
"Node": "x52",
"CheckID": "service:_nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001-sidecar-proxy:2",
"Name": "Connect Sidecar Listening",
"Status": "passing",
"Notes": "",
"Output": "TCP connect 192.168.88.252:31924: Success",
"ServiceID": "_nomad-task-0cc928cd-364f-b257-1e62-ef1b8ff25de7-group-api-count-api-9001-sidecar-proxy",
"ServiceName": "count-api-sidecar-proxy",
"ServiceTags": null,
"Type": "tcp",
"Interval": "10s",
"Timeout": "",
"ExposedPort": 0,
"Definition": {},
"Partition": "default",
"Namespace": "default",
"CreateIndex": 40,
"ModifyIndex": 46
}
]
}
],
"DNS": {
"TTL": ""
},
"Datacenter": "dc1",
"Failovers": 0,
"Index": 1,
"LastContact": 0,
"KnownLeader": true,
"ConsistencyLevel": "leader",
"NotModified": false,
"Backend": 0,
"ResultsFilteredByACLs": false
}
```

Looking at the service definition using service API

output

```
"_nomad-task-df294062-b346-d252-40f2-9110615a4080-group-dashboard-count-dashboard-9002-sidecar-proxy": {
"Kind": "connect-proxy",
"ID": "_nomad-task-df294062-b346-d252-40f2-9110615a4080-group-dashboard-count-dashboard-9002-sidecar-proxy",
"Service": "count-dashboard-sidecar-proxy",
"Tags": [],
"Meta": {
"external-source": "nomad"
},
"Port": 26732,
"Address": "192.168.88.252",
"TaggedAddresses": {
"consul-virtual": {
"Address": "240.0.0.1",
"Port": 26732
},
"lan_ipv4": {
"Address": "192.168.88.252",
"Port": 26732
},
"wan_ipv4": {
"Address": "192.168.88.252",
"Port": 26732
}
},
"Weights": {
"Passing": 1,
"Warning": 1
},
"EnableTagOverride": false,
"Proxy": {
"DestinationServiceName": "count-dashboard",
"DestinationServiceID": "_nomad-task-df294062-b346-d252-40f2-9110615a4080-group-dashboard-count-dashboard-9002",
"LocalServiceAddress": "127.0.0.1",
"LocalServicePort": 9002,
"Config": {
"bind_address": "0.0.0.0",
"bind_port": 26732,
"envoy_stats_tags": [
"nomad.alloc_id=df294062-b346-d252-40f2-9110615a4080"
]
},
"Upstreams": [
{
"DestinationType": "prepared_query",
"DestinationPartition": "default",
"DestinationNamespace": "default",
"DestinationName": "my-query",
"LocalBindPort": 8080,
"MeshGateway": {}
}
],
"MeshGateway": {},
"Expose": {}
},
"Namespace": "default",
"Partition": "default",
"Datacenter": "dc1"
}
```

That `Upstreams` block looks like the Nomad bits are working, passing the `destination_type` and query name to Consul.

```
"Upstreams": [
{
"DestinationType": "prepared_query",
"DestinationPartition": "default",
"DestinationNamespace": "default",
"DestinationName": "my-query",
"LocalBindPort": 8080,
"MeshGateway": {}
}
```

I saw this bit in the envoy config, I dunno if that means anything

```
{
"name": "prepared_query:my-query:127.0.0.1:8080",
"version": "94794baa248f6386e14df4ded1192bb8adc58172a13801b20e21ffb424d8c8cc",
"resource": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "prepared_query:my-query:127.0.0.1:8080",
"address": {
"socketAddress": {
"address": "127.0.0.1",
"portValue": 8080
}
},
"filterChains": [
{
"filters": [
{
"name": "envoy.filters.network.tcp_proxy",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"statPrefix": "upstream.prepared_query_my-query",
"cluster": "my-query.default.dc1.query.afab674c-7a2f-d612-f687-25d2cc8971e9.consul"
}
}
]
}
],
"trafficDirection": "OUTBOUND"
}
```

Note that if I run the same job but replace the prepared query with a normal service name, it works

```
destination_name = "count-api"
# destination_name = "my-query"
# destination_type = "prepared_query"
```