hashicorp / hashicorp/consul

Performance case for register service not scale

Open
#8,990 2 comments 0 reactions 0 assignees View on GitHub
theme/internals theme/performance type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

#### Overview of the Issue

when same service call register API, the QPS will be low and the mean response time for API would be seconds

Is it designed for this use case ?

```
# cat payload.json
{
"ID": "redis1",
"Name": "redis",
"Tags": ["primary", "v1"],
"Address": "127.0.0.1",
"Port": 8000,
"Meta": {
"redis_version": "4.0"
},
"EnableTagOverride": false,
"Weights": {
"Passing": 10,
"Warning": 1
}
}
# ab -n 500 -c 50 -T 'application/json' -u payload.json http://127.0.0.1:8500/v1/agent/service/register?replace-existing-checks=true

Requests per second: 26.51 [#/sec] (mean)
Time per request: 1885.854 [ms] (mean)
```

Use pprof , found the time spent in s.agent.State.SyncChanges();

Is it right here ? What's the performance considration for register service ?

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. ab run above command

### Consul info for both Client and Server

Client info

```
output from client 'consul info' command here
```

Server info

```
agent:
check_monitors = 0
check_ttls = 463
checks = 464
services = 465
build:
prerelease =
revision = 1200f25e
version = 1.6.2
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 127.0.0.1:8300
server = true
raft:
applied_index = 1229
commit_index = 1229
fsm_pending = 0
last_contact = 0
last_log_index = 1229
last_log_term = 2
last_snapshot_index = 0
last_snapshot_term = 0
latest_configuration = [{Suffrage:Voter ID:0b7026b6-777b-9cbb-9162-f63c7ad12589 Address:127.0.0.1:8300}]
latest_configuration_index = 1
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 8
goroutines = 545
max_procs = 8
os = linux
version = go1.12.13
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 1
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Linux run enrionment ,AMD64 arch

### Log Fragments

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported benchmark against the service registration API using the provided payload and Apache Benchmark command, then capture a pprof profile. Start with the reported s.agent.State.SyncChanges() path and compare behavior with the Consul 1.6.2 environment details. Done means determining whether the latency is expected and documenting or fixing the identified performance problem.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.