consul connect proxy fails with `[ERROR] proxy.upstream: failed to dial: error="tls: no certificates configured"`

#### Overview of the Issue

We are slowly migrating our system to nomad and consul. The way it currently works is that our main reverse proxy is not consul aware but it has defined upstream backends on 127.0.0.1 with a hardcoded port (5080). To allow ingress into consul mesh network we tried to use ingress gateway, but it failed because of https://github.com/hashicorp/consul/issues/12258. As a fast alternative, we replaced the ingress gateway with simple executable `consul connect --upstream ... --service ...` that runs on the same host as our main proxy. In nomad it is more or less specified as:
```hcl
...
group "load-balancer" {
constraint {
attribute = "${attr.unique.hostname}"
value = "foo.bar.com"
}

task "proxy" {
driver = "exec"

config {
command = "consul"
args = ["connect", "proxy",
"--upstream", "foo-bar-api:5080",
"--service", "foo-bar-loadbalancer"]
}
}
}
```
It works most of the time, but from time to time during deployment of the job, `consul connect proxy` fails to deliver. Job log is full of `[ERROR] proxy.upstream: failed to dial: error="tls: no certificates configured"`. The problem sometimes resolves itself in some minutes, but restarting consul (`systemctl restart consul`) is also "solving" it. Then logs stabilize with:
```
2022-02-07T13:58:56.405Z [ERROR] proxy.upstream: failed to dial: error="tls: no certificates configured"
2022-02-07T13:58:57.298Z [INFO] proxy: Proxy loaded config and ready to serve
2022-02-07T13:58:57.299Z [INFO] proxy: Parsed TLS identity: uri=spiffe://7ba92302-0ecf-f5d5-2ece-21fb63c51f58.consul/ns/default/dc/dc1/svc/foo-bar-loadbalancer roots=[pri-12lscpy.consul.ca.7ba92302.consul]
```
The annoying part is that as we run through "exec" nomad thinks that everything is ok, while our main proxy ships a lot of 502.

#### Reproduction Steps

Hard to reproduce reliably. Up to now it happens once per ~10 deploys.

### Consul info for both Client and Server

consul info:
```
agent:
check_monitors = 0
check_ttls = 0
checks = 47
services = 43
build:
prerelease =
revision = 37c7d06b
version = 1.11.2
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 100.xxx.xxx.xx5:8300
server = true
raft:
applied_index = 65200
commit_index = 65200
fsm_pending = 0
last_contact = 69.106946ms
last_log_index = 65200
last_log_term = 20
last_snapshot_index = 49156
last_snapshot_term = 13
latest_configuration = [{Suffrage:Voter ID:480f887b-354b-d91e-0946-49e8dd4d9681 Address:100.xxx.xxx.xx2:8300} {Suffrage:Voter ID:8f372f40-bebc-1a1e-1201-53115d44707a Address:100.xxx.xxx.xx9:8300} {Suffrage:Voter ID:c31c9067-9119-4491-f47d-f8432f168299 Address:100.xxx.xxx.x62:8300} {Suffrage:Voter ID:4e1f0be5-c3dd-1a2d-78f8-64fe74caa6ca Address:100.xxx.xxx.xx5:8300}]
latest_configuration_index = 0
num_peers = 3
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 20
runtime:
arch = amd64
cpu_count = 16
goroutines = 651
max_procs = 16
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 10
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 6
members = 4
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 7
members = 4
query_queue = 0
query_time = 1

```

### Operating system and Environment details

Amd64 Ubuntu 20.04

### Log Fragments

I tried to debug the issue a bit and I think I extracted the most important parts from pretty long logs. If something is missing feel free to ask.

After a lot of de-registrations:
```
Deregistered service: service=_nomad-task-0d4289e0-7794-b710-6951-a01c426dc8b9-group-....
```
There are some warnings about wrong agent state:
```
handling error in Cache.Notify: cache-type=service-http-checks error="Internal cache failure: service '_nomad-task-c5bea209-125b-73ce-4e1d-4bc3bbde9705-group-foo-bar-api-8000' not in agent state" index=0
```
and errors related to the previous warning:
```
Failed to handle update from watch: kind=connect-proxy proxy=_nomad-task-c5bea209-125b-73ce-4e1d-4bc3bbde9705-group-api-foo-bar-api-8000-sidecar-proxy service_id=_nomad-task-c5bea209-125b-73ce-4e1d-4bc3bbde9705-group-api-foo-bar-api-8000-sidecar-proxy id=service-http-checks:_nomad-task-c5bea209-125b-73ce-4e1d-4bc3bbde9705-group-api-foo-bar-api-8000 error="error filling agent cache: Internal cache failure: service '_nomad-task-c5bea209-125b-73ce-4e1d-4bc3bbde9705-group-api-foo-bar-api-8000' not in agent state"
```
Finally there is something classified as info, but related to the load balancer group and certificates, feels like the main issue:
```
Request cancelled: method=GET url=/v1/agent/connect/ca/leaf/foo-bar-loadbalancer from=127.0.0.1:50760 error="context canceled"
```
and
```
Request cancelled: method=GET url=/v1/agent/connect/ca/roots?index=9 from=127.0.0.1:50762 error="context canceled"
```
In couple of seconds later other consul servers are reporting:
```
[ERROR] agent.envoy: Error receiving new DeltaDiscoveryRequest; closing request channel: error="rpc error: code = Canceled desc = context canceled"
```