Race condition highlighted in TestAgent_Service

On upstream 3956cff60fa6c67c8996c168058c9bc2d6dc5a31 as show as failures in https://circleci.com/gh/hashicorp/consul/164453

```
i=0; while go test -timeout 30s github.com/hashicorp/consul/agent -run '^(TestAgent_Service)$'; do go clean -testcache; i=$((i + 1)); printf "$i "; done
ok github.com/hashicorp/consul/agent (cached)
1 ok github.com/hashicorp/consul/agent 1.761s
2 ok github.com/hashicorp/consul/agent 1.759s
3 ok github.com/hashicorp/consul/agent 1.714s
4 ok github.com/hashicorp/consul/agent 1.756s
5 ok github.com/hashicorp/consul/agent 1.819s
6 ok github.com/hashicorp/consul/agent 1.805s
7 ok github.com/hashicorp/consul/agent 1.713s
8 ok github.com/hashicorp/consul/agent 1.822s
9 ok github.com/hashicorp/consul/agent 1.789s
10 ok github.com/hashicorp/consul/agent 1.724s
11 ok github.com/hashicorp/consul/agent 1.801s
12 ok github.com/hashicorp/consul/agent 1.815s
13 ok github.com/hashicorp/consul/agent 1.823s
14 ok github.com/hashicorp/consul/agent 1.765s
15 ok github.com/hashicorp/consul/agent 1.713s
16 ok github.com/hashicorp/consul/agent 1.698s
17 ok github.com/hashicorp/consul/agent 1.688s
18 ok github.com/hashicorp/consul/agent 1.749s
19 ok github.com/hashicorp/consul/agent 1.832s
20 ok github.com/hashicorp/consul/agent 1.882s
21 ok github.com/hashicorp/consul/agent 1.706s
22 ok github.com/hashicorp/consul/agent 1.729s
23 ok github.com/hashicorp/consul/agent 1.683s
24 ok github.com/hashicorp/consul/agent 1.753s
25 ok github.com/hashicorp/consul/agent 1.744s
26 ok github.com/hashicorp/consul/agent 1.721s
27 ok github.com/hashicorp/consul/agent 1.738s
28 ok github.com/hashicorp/consul/agent 1.724s
29 ok github.com/hashicorp/consul/agent 2.066s
30 ok github.com/hashicorp/consul/agent 1.730s
31 ok github.com/hashicorp/consul/agent 1.730s
32 ok github.com/hashicorp/consul/agent 1.732s
33 ok github.com/hashicorp/consul/agent 1.783s
34 ok github.com/hashicorp/consul/agent 1.731s
35 ok github.com/hashicorp/consul/agent 1.707s
36 ok github.com/hashicorp/consul/agent 1.765s
37 [INFO] freeport: blockSize 1500 too big for system limit 256. Adjusting...
[INFO] freeport: detected ephemeral port range of [49152, 65535]
31:51.550 [WARN] TestAgent: The 'acl_datacenter' field is deprecated. Use the 'primary_datacenter' field instead.
31:51.550 [WARN] TestAgent: bootstrap = true: do not enable unless necessary
31:51.701 [INFO] TestAgent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:b3209756-abf1-00a7-1b9a-2102d250d335 Address:127.0.0.1:17090}]"
31:51.701 [INFO] TestAgent.server.raft: entering follower state: follower="Node at 127.0.0.1:17090 [Follower]" leader=
31:51.703 [INFO] TestAgent.server.serf.wan: serf: EventMemberJoin: Node-b3209756-abf1-00a7-1b9a-2102d250d335.dc1 127.0.0.1
31:51.703 [INFO] TestAgent.server.serf.lan: serf: EventMemberJoin: Node-b3209756-abf1-00a7-1b9a-2102d250d335 127.0.0.1
31:51.703 [INFO] TestAgent.server: Handled event for server in area: event=member-join server=Node-b3209756-abf1-00a7-1b9a-2102d250d335.dc1 area=wan
31:51.703 [INFO] TestAgent.server: Adding LAN server: server="Node-b3209756-abf1-00a7-1b9a-2102d250d335 (Addr: tcp/127.0.0.1:17090) (DC: dc1)"
31:51.704 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17085 network=tcp
31:51.704 [INFO] TestAgent: Started DNS server: address=127.0.0.1:17085 network=udp
31:51.704 [INFO] TestAgent: Started HTTP server: address=127.0.0.1:17086 network=tcp
31:51.705 [INFO] TestAgent: started state syncer
31:51.705 [INFO] TestAgent: Started gRPC server: address=127.0.0.1:17091 network=tcp
31:51.755 [DEBUG] TestAgent.server: Cannot upgrade to new ACLs: leaderMode=3 mode=2 found=true leader=
31:51.768 [WARN] TestAgent.server.raft: heartbeat timeout reached, starting election: last-leader=
31:51.768 [INFO] TestAgent.server.raft: entering candidate state: node="Node at 127.0.0.1:17090 [Candidate]" term=2
31:51.808 [DEBUG] TestAgent.server.raft: votes: needed=1
31:51.808 [DEBUG] TestAgent.server.raft: vote granted: from=b3209756-abf1-00a7-1b9a-2102d250d335 term=2 tally=1
31:51.808 [INFO] TestAgent.server.raft: election won: tally=1
31:51.808 [INFO] TestAgent.server.raft: entering leader state: leader="Node at 127.0.0.1:17090 [Leader]"
31:51.808 [INFO] TestAgent.server: cluster leadership acquired
31:51.808 [INFO] TestAgent.server: New leader elected: payload=Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:51.836 [INFO] TestAgent.server: initializing acls
31:51.849 [INFO] TestAgent.server: Created ACL 'global-management' policy
31:51.849 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated
31:51.856 [INFO] TestAgent.server: initializing acls
31:51.856 [WARN] TestAgent.server: Configuring a non-UUID master token is deprecated
31:51.883 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration
31:51.883 [INFO] TestAgent.server: Bootstrapped ACL master token from configuration
31:51.897 [INFO] TestAgent.server: Created ACL anonymous token from configuration
31:51.897 [INFO] TestAgent.leader: started routine: routine="legacy ACL token upgrade"
31:51.897 [INFO] TestAgent.leader: started routine: routine="acl token reaping"
31:51.897 [DEBUG] TestAgent.server: transitioning out of legacy ACL mode
31:51.898 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:51.898 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b3209756-abf1-00a7-1b9a-2102d250d335.dc1
31:51.898 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b3209756-abf1-00a7-1b9a-2102d250d335.dc1 area=wan
31:51.898 [INFO] TestAgent.server: Updating LAN server: server="Node-b3209756-abf1-00a7-1b9a-2102d250d335 (Addr: tcp/127.0.0.1:17090) (DC: dc1)"
31:51.911 [INFO] TestAgent.server: Created ACL anonymous token from configuration
31:51.911 [INFO] TestAgent.server.serf.lan: serf: EventMemberUpdate: Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:51.911 [INFO] TestAgent.server.serf.wan: serf: EventMemberUpdate: Node-b3209756-abf1-00a7-1b9a-2102d250d335.dc1
31:51.911 [INFO] TestAgent.server: Handled event for server in area: event=member-update server=Node-b3209756-abf1-00a7-1b9a-2102d250d335.dc1 area=wan
31:51.911 [INFO] TestAgent.server: Updating LAN server: server="Node-b3209756-abf1-00a7-1b9a-2102d250d335 (Addr: tcp/127.0.0.1:17090) (DC: dc1)"
31:51.952 [DEBUG] connect.ca.consul: consul CA provider configured: id=07:80:c8:de:f6:41:86:29:8f:9c:b8:17:d6:48:c2:d5:c5:5c:7f:0c:03:f7:cf:97:5a:a7:c1:68:aa:23:ae:81 is_primary=true
31:52.000 [INFO] TestAgent.server.connect: initialized primary datacenter CA with provider: provider=consul
31:52.000 [INFO] TestAgent.leader: started routine: routine="federation state anti-entropy"
31:52.000 [INFO] TestAgent.leader: started routine: routine="federation state pruning"
31:52.000 [INFO] TestAgent.leader: started routine: routine="CA root pruning"
31:52.001 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:52.001 [INFO] TestAgent.server: member joined, marking health alive: member=Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:52.015 [INFO] TestAgent.server: federation state anti-entropy synced
31:52.028 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:52.028 [DEBUG] TestAgent.server: Skipping self join check for node since the cluster is too small: node=Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:52.041 [DEBUG] TestAgent.acl: dropping node from result due to ACLs: node=Node-b3209756-abf1-00a7-1b9a-2102d250d335
31:52.043 [ERROR] TestAgent.proxycfg: watch error: id=service-http-checks:web1 error="invalid type for service checks response: cache.FetchResult, want: []structs.CheckType"
31:52.096 [INFO] TestAgent: Synced node info
31:52.096 [WARN] TestAgent: Service registration blocked by ACLs: service=web accessorID=
31:52.109 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.109 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.109 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth
31:52.109 [DEBUG] TestAgent: Node info in sync
31:52.109 [WARN] TestAgent: Service registration blocked by ACLs: service=web accessorID=
31:52.122 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.122 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.122 [DEBUG] TestAgent: Node info in sync
31:52.122 [DEBUG] TestAgent: Service in sync: service=web
31:52.122 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:52.122 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.127 [WARN] TestAgent: Check socket connection failed: check=service:web-sidecar-proxy error="dial tcp 127.0.0.1:8000: connect: connection refused"
31:52.127 [WARN] TestAgent: Check is now critical: check=service:web-sidecar-proxy
31:52.139 [DEBUG] TestAgent: Node info in sync
31:52.139 [DEBUG] TestAgent: Service in sync: service=web
31:52.160 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.160 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.175 [DEBUG] TestAgent: Node info in sync
31:52.175 [DEBUG] TestAgent: Service in sync: service=web
31:52.188 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.188 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.311 [DEBUG] TestAgent: Node info in sync
31:52.311 [DEBUG] TestAgent: Service in sync: service=web
31:52.324 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.324 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.340 [DEBUG] TestAgent: Node info in sync
31:52.340 [DEBUG] TestAgent: Service in sync: service=web
31:52.353 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.353 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.454 [ERROR] TestAgent.proxycfg: watch error: id=service-http-checks:web1 error="invalid type for service checks response: cache.FetchResult, want: []structs.CheckType"
31:52.470 [DEBUG] TestAgent: Node info in sync
31:52.470 [DEBUG] TestAgent: Service in sync: service=web
31:52.484 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.484 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.485 [ERROR] TestAgent.proxycfg: watch error: id=service-http-checks:web1 error="invalid type for service checks response: cache.FetchResult, want: []structs.CheckType"
31:52.501 [DEBUG] TestAgent: Node info in sync
31:52.501 [DEBUG] TestAgent: Service in sync: service=web
31:52.515 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.515 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.635 [DEBUG] TestAgent: Node info in sync
31:52.635 [DEBUG] TestAgent: Service in sync: service=web
31:52.650 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.650 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.746 [DEBUG] TestAgent: Node info in sync
31:52.746 [DEBUG] TestAgent: Service in sync: service=web
31:52.759 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:52.759 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.863 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy
31:52.863 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy
31:52.863 [DEBUG] TestAgent: removed service: service=web
31:52.865 [DEBUG] TestAgent: restored service definition from file: service=web-sidecar-proxy file=/var/folders/yz/ln_hlbcd1_bczjfwwptyzms9vyfn_2/T/TestAgent-agent224862118/services/252ecd1f329632e74a20f35a05bc80fd
31:52.865 [ERROR] TestAgent.proxycfg: watch error: id=service-http-checks:web1 error="invalid type for service checks response: cache.FetchResult, want: []structs.CheckType"
31:52.865 [DEBUG] TestAgent: restored health check from file: check=service:web-sidecar-proxy file=/var/folders/yz/ln_hlbcd1_bczjfwwptyzms9vyfn_2/T/TestAgent-agent224862118/checks/88674fc18775aada5bb6be7cf8c28049
31:52.871 [DEBUG] TestAgent: Skipping remote check since it is managed automatically: check=serfHealth
31:52.871 [DEBUG] TestAgent: Node info in sync
31:52.871 [WARN] TestAgent: Service registration blocked by ACLs: service=web accessorID=
31:52.871 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:52.871 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.871 [DEBUG] TestAgent: Node info in sync
31:52.871 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:52.871 [DEBUG] TestAgent: Service in sync: service=web
31:52.871 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:52.989 [DEBUG] TestAgent: Node info in sync
31:52.989 [DEBUG] TestAgent: Service in sync: service=web
31:53.008 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:53.008 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.008 [DEBUG] TestAgent: Node info in sync
31:53.008 [DEBUG] TestAgent: Service in sync: service=web
31:53.008 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:53.008 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.109 [DEBUG] TestAgent: removed service: service=web
31:53.109 [DEBUG] TestAgent: removed check: check=service:web-sidecar-proxy
31:53.109 [DEBUG] TestAgent: removed service: service=web-sidecar-proxy
31:53.110 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.110 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=service-http-checks:web1 error="invalid type for service checks response: cache.FetchResult, want: []structs.CheckType"
31:53.110 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=intentions error="error filling agent cache: Permission denied"
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=discovery-chain:db error="error filling agent cache: Permission denied"
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=discovery-chain:db error="error filling agent cache: Permission denied"
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=discovery-chain:db error="error filling agent cache: Permission denied"
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=discovery-chain:db error="error filling agent cache: Permission denied"
31:53.110 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=intentions error="error filling agent cache: Permission denied"
31:53.110 [ERROR] TestAgent.proxycfg: watch error: id=intentions error="error filling agent cache: Permission denied"
31:53.111 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.111 [ERROR] TestAgent.proxycfg: watch error: id=leaf error="error filling agent cache: Permission denied"
31:53.111 [ERROR] TestAgent.proxycfg: watch error: id=intentions error="error filling agent cache: Permission denied"
31:53.111 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.111 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.111 [WARN] TestAgent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=web accessorID=
31:53.111 [ERROR] TestAgent.proxycfg: watch error: id=leaf error="error filling agent cache: Permission denied"
31:53.111 [DEBUG] TestAgent: service exists, not restoring from file: service=web-sidecar-proxy file=/var/folders/yz/ln_hlbcd1_bczjfwwptyzms9vyfn_2/T/TestAgent-agent224862118/services/252ecd1f329632e74a20f35a05bc80fd
31:53.111 [ERROR] TestAgent.proxycfg: watch error: id=leaf error="error filling agent cache: Permission denied"
31:53.111 [ERROR] TestAgent.proxycfg: watch error: id=leaf error="error filling agent cache: Permission denied"
31:53.111 [DEBUG] TestAgent: check exists, not restoring from file: check=service:web-sidecar-proxy file=/var/folders/yz/ln_hlbcd1_bczjfwwptyzms9vyfn_2/T/TestAgent-agent224862118/checks/88674fc18775aada5bb6be7cf8c28049
31:53.112 [DEBUG] TestAgent: Node info in sync
31:53.112 [WARN] TestAgent: Service registration blocked by ACLs: service=web accessorID=
31:53.112 [WARN] TestAgent: Service registration blocked by ACLs: service=web-sidecar-proxy accessorID=
31:53.112 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.113 [ERROR] TestAgent.proxycfg: watch error: id=service-http-checks:web1 error="invalid type for service checks response: cache.FetchResult, want: []structs.CheckType"
31:53.150 [DEBUG] TestAgent: Node info in sync
31:53.150 [DEBUG] TestAgent: Service in sync: service=web
31:53.185 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:53.185 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.185 [DEBUG] TestAgent: Node info in sync
31:53.185 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:53.185 [DEBUG] TestAgent: Service in sync: service=web
31:53.185 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.220 [DEBUG] TestAgent: Node info in sync
31:53.247 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:53.247 [DEBUG] TestAgent: Service in sync: service=web
31:53.247 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.248 [DEBUG] TestAgent: Node info in sync
31:53.248 [DEBUG] TestAgent: Service in sync: service=web
31:53.248 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:53.248 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.329 [DEBUG] TestAgent: Node info in sync
31:53.329 [DEBUG] TestAgent: Service in sync: service=web
31:53.359 [INFO] TestAgent: Synced service: service=web-sidecar-proxy
31:53.359 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.359 [DEBUG] TestAgent: Node info in sync
31:53.359 [DEBUG] TestAgent: Service in sync: service=web
31:53.359 [DEBUG] TestAgent: Service in sync: service=web-sidecar-proxy
31:53.359 [DEBUG] TestAgent: Check in sync: check=service:web-sidecar-proxy
31:53.402 [INFO] TestAgent: Requesting shutdown
31:53.402 [INFO] TestAgent.server: shutting down server
31:53.403 [DEBUG] TestAgent.leader: stopping routine: routine="federation state pruning"
31:53.403 [DEBUG] TestAgent.leader: stopping routine: routine="CA root pruning"
31:53.403 [DEBUG] TestAgent.leader: stopping routine: routine="legacy ACL token upgrade"
31:53.403 [DEBUG] TestAgent.leader: stopping routine: routine="acl token reaping"
31:53.403 [DEBUG] TestAgent.leader: stopping routine: routine="federation state anti-entropy"
31:53.403 [DEBUG] TestAgent.leader: stopped routine: routine="CA root pruning"
31:53.403 [DEBUG] TestAgent.leader: stopped routine: routine="acl token reaping"
31:53.403 [DEBUG] TestAgent.leader: stopped routine: routine="legacy ACL token upgrade"
31:53.403 [DEBUG] TestAgent.leader: stopped routine: routine="federation state pruning"
31:53.403 [WARN] TestAgent.server.serf.lan: serf: Shutdown without a Leave
31:53.421 [WARN] TestAgent.server.serf.wan: serf: Shutdown without a Leave
31:53.441 [INFO] TestAgent.server.router.manager: shutting down
31:53.442 [INFO] TestAgent: consul server down
31:53.442 [INFO] TestAgent: shutdown complete
31:53.442 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17085 network=tcp
31:53.442 [INFO] TestAgent: Stopping server: protocol=DNS address=127.0.0.1:17085 network=udp
31:53.442 [INFO] TestAgent: Stopping server: protocol=HTTP address=127.0.0.1:17086 network=tcp
31:53.442 [INFO] TestAgent: Waiting for endpoints to shut down
31:53.442 [INFO] TestAgent: Endpoints down
--- FAIL: TestAgent_Service (1.91s)
--- FAIL: TestAgent_Service/simple_fetch_-_proxy (0.08s)
agent_endpoint_test.go:603:
Error Trace: agent_endpoint_test.go:603
Error: Should be true
Test: TestAgent_Service/simple_fetch_-_proxy
Messages: should not have waited, took 13.290074ms
FAIL
FAIL github.com/hashicorp/consul/agent 1.947s
FAIL
```