hashicorp / hashicorp/consul

Consistent Reads after update to version 1.8

Open
#9,879 8 comments 0 reactions 1 assignee Claimed by @jsosulska View on GitHub
theme/telemetry type/bug type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

After updating from version 1.7.2 to 1.8.8 I noticed that the number of consistentRead has increased (prior was 0).

Screen Shot 2021-03-15 at 1 32 28 PM

I can replicate the issue starting from version 1.8.0.
I tested also latest version 1.9.4 and the issue is there as well.
I attempted to look at all PRs from version 1.7.13 to 1.8.0 but didn't find anything obvious.
Opening this issue hoping to get some feedbacks and help for troubleshooting.

I was able to replicate the issue locally with the following steps:

### Create basic config file to enable prom metrics
```
cat < consul-config.json
{
"telemetry": {
"disable_hostname": true,
"prometheus_retention_time": "60s"
}
}
EOF
```
### Execute the following cmds:
```
CONSUL_VERSION=1.7.13
docker run -d --rm --name=consul-0 -p 8500:8500 -p 8600:8600 -v $(pwd)/consul-config.json:/consul/config/consul-config.json -e CONSUL_BIND_INTERFACE=eth0 consul:${CONSUL_VERSION}
JOIN_IP=$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' consul-0)

for i in $(seq 4)
do
docker run -d --rm --name=consul-${i} -p 850${i}:8500 -p 860${i}:8600 -v $(pwd)/consul-config.json:/consul/config/consul-config.json -e CONSUL_BIND_INTERFACE=eth0 consul:${CONSUL_VERSION}
done

for i in $(seq 4)
do
docker exec consul-${i} consul join ${JOIN_IP}
done

curl http://127.0.0.1:8500/v1/agent/metrics?format=prometheus -sS | grep -i consisten
docker rm -f consul-0 consul-1 consul-2 consul-3 consul-4
```

The `grep` will not return any results, now test with version 1.8.0:
```
CONSUL_VERSION=1.8.0
docker run -d --rm --name=consul-0 -p 8500:8500 -p 8600:8600 -v $(pwd)/consul-config.json:/consul/config/consul-config.json -e CONSUL_BIND_INTERFACE=eth0 consul:${CONSUL_VERSION}
JOIN_IP=$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' consul-0)

for i in $(seq 4)
do
docker run -d --rm --name=consul-${i} -p 850${i}:8500 -p 860${i}:8600 -v $(pwd)/consul-config.json:/consul/config/consul-config.json -e CONSUL_BIND_INTERFACE=eth0 consul:${CONSUL_VERSION}
done

for i in $(seq 4)
do
docker exec consul-${i} consul join ${JOIN_IP}
done

curl http://127.0.0.1:8500/v1/agent/metrics?format=prometheus -sS | grep -i consisten
docker rm -f consul-0 consul-1 consul-2 consul-3 consul-4
```

The grep will return something like:
```
curl http://127.0.0.1:8500/v1/agent/metrics?format=prometheus -sS | grep -i consisten
# HELP consul_rpc_consistentRead consul_rpc_consistentRead
# TYPE consul_rpc_consistentRead summary
consul_rpc_consistentRead_sum 0.820200003683567
consul_rpc_consistentRead_count 6
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.