apple / apple/foundationdb

Drift between `total_queries` and `finished_queries` metrics in status when setting watches from multiple clients

Open
#13,796 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

If you run two parallel instances of [this test program](https://gist.github.com/jkt-signal/1c3ea2a4a6aafd51571b462c57b50fd9) against a fresh fdb database and then, any amount of time later, check its `status.json`, you'll see something like this snippet:

```json
"cluster": {
"processes": {
"xyz": {
"roles": [
{
"role": "storage",
"total_queries": {
"counter": 3082050,
"hz": 15.1999,
"roughness": 6.7680300000000004
},
"finished_queries": {
"counter": 3082015,
"hz": 15.1999,
"roughness": 6.7680300000000004
},
"query_queue_max": 41,
}
]
}
}
}
```

This difference between `total_queries` and `finished_queries` will remain elevated for the life of the storage process (and will get larger every time you run two instances of the test program). `query_queue_max`, since it is computed based on the difference between those two counters, will likewise go up and never come back down.

This is fixed in #13786 .

Contributor guide

Open the contributing guide

Research direction

Start with the linked test program and a fresh FoundationDB database, then inspect status.json while running two instances to reproduce the counter drift. Read #13786 to understand the related fix; done means total_queries and finished_queries remain aligned and query_queue_max no longer stays elevated.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.