X-Consul-Index going backwards due to different results from servers

#### Overview of the Issue
We are using blocking queries to watch for health changes in our services, applying the standard mechanism of polling with some index and using the returned X-Consul-Index header as the index for the next poll.
We query the local consul agent with a query that looks like (note the stale=true):
v1/health/service/ServiceName?index=&passing=true&wait=1s&stale=true&tag=environment-prod
We recently found out that each server is consistently returning the same index (can be for many hours, until the watched service changes), but different servers are returning different indexes.
Since we are using stale=true, this is causing the blocking queries to return X-Consul-Index that keeps changing up and down according to which consul server was queried.
Is this issue expected? Is the combination of stale + blocking queries not supported?

`http://Follower1:8500/v1/health/service/ServiceName?index=0&passing=true&wait=1s&stale=true
Vary: Accept-Encoding
X-Consul-Effective-Consistency: stale
X-Consul-Index: 1672207871
X-Consul-Knownleader: true
X-Consul-Lastcontact: 34`

`http://Follower2:8500/v1/health/service/ServiceName?index=0&passing=true&wait=1s&stale=true
Vary: Accept-Encoding
X-Consul-Effective-Consistency: stale
X-Consul-Index: 1672201982
X-Consul-Knownleader: true
X-Consul-Lastcontact: 7`

`http://Leader:8500/v1/health/service/ServiceName?index=0&passing=true&wait=1s&stale=true
Vary: Accept-Encoding
X-Consul-Effective-Consistency: stale
X-Consul-Index: 1672198845
X-Consul-Knownleader: true
X-Consul-Lastcontact: 0`

When we remove the `stale=true` parameter we always get "1672198845" which is the index returned by the leader.
We also noticed that for the problematic services in this state, the `ModifiedIndex` and `CreateIndex` are the same and identical to the value returned in the header (then setting stale=true)

### Consul info Follower1

Follower1

```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
bootstrap = false
known_datacenters = 4
leader = false
leader_addr = :8300
server = true
raft:
applied_index = 1677031174
commit_index = 1677031174
fsm_pending = 0
last_contact = 61.142668ms
last_log_index = 1677031175
last_log_term = 359458
last_snapshot_index = 1677031112
last_snapshot_term = 359458
latest_configuration = [{Suffrage:Voter ID:dde09378-4944-7018-9711-6be17deb00f3 Address::8300} {Suffrage:Voter ID:43b09dd3-b385-3b71-b4c5-9ef03df42ac6 Address::8300} {Suffrage:Voter ID:cfdd42a9-44e5-590a-1a50-e5e79935b30a Address::8300}]
latest_configuration_index = 1669000507
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 359458
runtime:
arch = amd64
cpu_count = 96
goroutines = 86511
max_procs = 96
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 154693
failed = 2
health_score = 0
intent_queue = 0
left = 7
member_time = 241034
members = 2684
query_queue = 0
query_time = 1294
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2042
members = 10
query_queue = 0
query_time = 1
```

### Consul info Follower2

Follower2

```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
bootstrap = false
known_datacenters = 4
leader = false
leader_addr = :8300
server = true
raft:
applied_index = 1677031419
commit_index = 1677031419
fsm_pending = 0
last_contact = 14.931091ms
last_log_index = 1677031419
last_log_term = 359458
last_snapshot_index = 1677016577
last_snapshot_term = 359458
latest_configuration = [{Suffrage:Voter ID:dde09378-4944-7018-9711-6be17deb00f3 Address::8300} {Suffrage:Voter ID:43b09dd3-b385-3b71-b4c5-9ef03df42ac6 Address::8300} {Suffrage:Voter ID:cfdd42a9-44e5-590a-1a50-e5e79935b30a Address::8300}]
latest_configuration_index = 1669000507
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 359458
runtime:
arch = amd64
cpu_count = 96
goroutines = 90189
max_procs = 96
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 154693
failed = 2
health_score = 0
intent_queue = 0
left = 7
member_time = 241034
members = 2684
query_queue = 0
query_time = 1294
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2042
members = 10
query_queue = 0
query_time = 1
```

### Consul info Leader

Leader info

```
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 39f93f01
version = 1.2.1
consul:
bootstrap = false
known_datacenters = 4
leader = true
leader_addr = :8300
server = true
raft:
applied_index = 1677031859
commit_index = 1677031859
fsm_pending = 0
last_contact = 0
last_log_index = 1677031859
last_log_term = 359458
last_snapshot_index = 1677020696
last_snapshot_term = 359458
latest_configuration = [{Suffrage:Voter ID:dde09378-4944-7018-9711-6be17deb00f3 Address::8300} {Suffrage:Voter ID:43b09dd3-b385-3b71-b4c5-9ef03df42ac6 Address::8300} {Suffrage:Voter ID:cfdd42a9-44e5-590a-1a50-e5e79935b30a Address::8300}]
latest_configuration_index = 1669000507
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 359458
runtime:
arch = amd64
cpu_count = 96
goroutines = 77999
max_procs = 96
os = linux
version = go1.10.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 154693
failed = 2
health_score = 0
intent_queue = 0
left = 7
member_time = 241034
members = 2684
query_queue = 0
query_time = 1294
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2042
members = 10
query_queue = 0
query_time = 1
```

### Operating system and Environment details
All servers are running ubuntu16.o4 and kernel version: 4.15.0-99-generic
Consul data dir is running on SSD, all servers have 96 cores

Most agents run on kube nodes(ubuntu16), services are running in containers and registered upon deployment using registrator.
Other agents are running on variety of OS distribution(all are ubuntu flavours) but service in this case are registered using json flies.