cockroachdb / cockroachdb/cockroach
drt: pickOperation has skewed pick rate when multiple sets exist
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Consider an especially expensive/long operation, e.g. `INSPECT DATABASE`, that we wish to run less frequently compared to other operations. Currently this can be done by setting a filter and setting a higher `--schedule`, e.g:
```
operations:
parallelism: 3
sets:
- filter: ^(?!inspect\/database$).*
cadence: 5m
- filter: ^inspect\/database$
cadence: 20m
```
However, if we inspect how `pickOperation` actually chooses random operations:
```
setIdx := rng.Intn(len(r.specs))
opSpecsSet := r.specs[setIdx]
opSpec := &opSpecsSet[rng.Intn(len(opSpecsSet))]
```
We see that we first randomly pick a set, not accounting for the size of the set, i.e. we will actually pick `inspect/database` 50% of the time and likely run the operation _more_, not less.
Instead, `pickOperation` should be changed to account for set size and pick all operations with even probability.
------
Tangentially related behavior that should also be reevaluated is how we block in `pickOperation` when we select an operation on cooldown (i.e. it hasn't been `--schedule` duration since the last run):
```
lastRun := r.mu.lastRun[opSpec.Name]
eligibleForNextRun := lastRun.Add(r.config.Operations.Sets[setIdx].Cadence)
if timeutil.Now().Compare(eligibleForNextRun) < 0 {
// Find another operation to run.
r.mu.completed.Wait()
return true
}
```
As seen above, we block on the `completed` `Sync.Cond`. However, the only place we ever `Broadcast` or `Signal` is at the end of `runOperation`. Consider a parallelism of 1, where we select the same operation back to back. Because there are no other operations concurrently running, we would block indefinitely on the `Wait()`. This can also be hit on > 1 parallelism given an unlucky enough scheduling, e.g. all workers are in the `pickOperation` method and select operations that are on cooldown or can't be run concurrently.
Jira issue: CRDB-57132
Contributor guide
Assessment
This issue has not been assessed yet.