Drift between `total_queries` and `finished_queries` metrics in status when setting watches from multiple clients
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
If you run two parallel instances of [this test program](https://gist.github.com/jkt-signal/1c3ea2a4a6aafd51571b462c57b50fd9) against a fresh fdb database and then, any amount of time later, check its `status.json`, you'll see something like this snippet:
```json
"cluster": {
"processes": {
"xyz": {
"roles": [
{
"role": "storage",
"total_queries": {
"counter": 3082050,
"hz": 15.1999,
"roughness": 6.7680300000000004
},
"finished_queries": {
"counter": 3082015,
"hz": 15.1999,
"roughness": 6.7680300000000004
},
"query_queue_max": 41,
}
]
}
}
}
```
This difference between `total_queries` and `finished_queries` will remain elevated for the life of the storage process (and will get larger every time you run two instances of the test program). `query_queue_max`, since it is computed based on the difference between those two counters, will likewise go up and never come back down.
This is fixed in #13786 .
Contributor guide
Research direction
Start with the linked test program and a fresh FoundationDB database, then inspect status.json while running two instances to reproduce the counter drift. Read #13786 to understand the related fix; done means total_queries and finished_queries remain aligned and query_queue_max no longer stays elevated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100