hashicorp / hashicorp/consul

Consul Catalog not respecting `-stale` flag

Open
#9,610 0 comments 0 reactions 0 assignees View on GitHub
needs-investigation theme/api type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

Hi folks, I'm noticing some odd behavior when trying to use the `-stale` flag with commands like `consul catalog services`, mainly to account for times when the leader goes down and there are queries during a new election. Can anyone also please just confirm if this is the correct use of allowing stale, or if another level of consistency should be used?

It seems that `consul catalog services -stale` still requires the request be handled by the leader, though. However, `curl localhost:8500/v1/catalog/services?stale` does not. I assumed these would behave the same way, but is there possibly a bug with the client?

Thanks folks!

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a cluster with 1 client node and 3 server nodes
2. On client, run
```bash
while true; do date; consul catalog services -stale; sleep 1; done
```
3. On the leader, run `pkill consul`
4. View failed requests on client while new leader is being elected

Note that `192.168.33.32` was the leader where the Consul process was killed, and `192.168.33.33` is another Consul server.

```
Thu Jan 21 17:16:25 UTC 2021
consul
generic
Thu Jan 21 17:16:26 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error making call: rpc error making call: EOF)
Thu Jan 21 17:16:27 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp 172.28.128.254:0->192.168.33.32:8300: connect: connection refused)
Thu Jan 21 17:16:29 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error making call: rpc error getting client: failed to get conn: dial tcp 192.168.33.33:0->192.168.33.32:8300: connect: connection refused)
Thu Jan 21 17:16:30 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp 172.28.128.254:0->192.168.33.32:8300: connect: connection refused)
Thu Jan 21 17:16:31 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp 172.28.128.254:0->192.168.33.32:8300: connect: connection refused)
Thu Jan 21 17:16:32 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error making call: No cluster leader)
Thu Jan 21 17:16:40 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error making call: No cluster leader)
Thu Jan 21 17:16:48 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error making call: No cluster leader)
Thu Jan 21 17:16:56 UTC 2021
Error listing services: Unexpected response code: 500 (rpc error making call: No cluster leader)
Thu Jan 21 17:17:04 UTC 2021
consul
generic
```

Alternatively, run this loop on the client when the new leader is killed:
```bash
while true; do date; curl localhost:8500/v1/catalog/services?stale; sleep 1; done
```

Note that there are no errors:
```
{"consul":[],"generic":["kitchen"]}Thu Jan 21 17:19:41 UTC 2021
{"consul":[],"generic":["kitchen"]}Thu Jan 21 17:19:42 UTC 2021
{"consul":[],"generic":["kitchen"]}Thu Jan 21 17:19:43 UTC 2021
{"consul":[],"generic":["kitchen"]}Thu Jan 21 17:19:44 UTC 2021
{"consul":[],"generic":["kitchen"]}Thu Jan 21 17:19:45 UTC 2021
{"consul":[],"generic":["kitchen"]}Thu Jan 21 17:19:46 UTC 2021
```

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 1
build:
prerelease =
revision = ca5c3894
version = 1.9.1
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 1
goroutines = 50
max_procs = 2
os = linux
version = go1.15.6
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 15
failed = 1
health_score = 0
intent_queue = 0
left = 2
member_time = 332
members = 5
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 1
build:
prerelease =
revision = ca5c3894
version = 1.9.1
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 192.168.33.32:8300
server = true
raft:
applied_index = 857
commit_index = 857
fsm_pending = 0
last_contact = 60.125966ms
last_log_index = 857
last_log_term = 39
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:edab1288-f08c-6c88-4dcf-8c5afc0c0865 Address:192.168.33.32:8300} {Suffrage:Voter ID:572be8c2-5d76-b3da-a98b-bb808bc8ce7e Address:192.168.33.33:8300}]
latest_configuration_index = 0
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 39
runtime:
arch = amd64
cpu_count = 1
goroutines = 91
max_procs = 2
os = linux
version = go1.15.6
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 16
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 333
members = 3
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 155
members = 2
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Ubuntu 18.04

### Log Fragments
See above.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.