cockroachdb / cockroachdb/cockroach

drt: pickOperation has skewed pick rate when multiple sets exist

Open
#158,149 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-foundations C-bug P-3 T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Consider an especially expensive/long operation, e.g. `INSPECT DATABASE`, that we wish to run less frequently compared to other operations. Currently this can be done by setting a filter and setting a higher `--schedule`, e.g:

```
operations:
parallelism: 3
sets:
- filter: ^(?!inspect\/database$).*
cadence: 5m
- filter: ^inspect\/database$
cadence: 20m
```

However, if we inspect how `pickOperation` actually chooses random operations:

```
setIdx := rng.Intn(len(r.specs))
opSpecsSet := r.specs[setIdx]
opSpec := &opSpecsSet[rng.Intn(len(opSpecsSet))]
```

We see that we first randomly pick a set, not accounting for the size of the set, i.e. we will actually pick `inspect/database` 50% of the time and likely run the operation _more_, not less.

Instead, `pickOperation` should be changed to account for set size and pick all operations with even probability.

------

Tangentially related behavior that should also be reevaluated is how we block in `pickOperation` when we select an operation on cooldown (i.e. it hasn't been `--schedule` duration since the last run):

```
lastRun := r.mu.lastRun[opSpec.Name]
eligibleForNextRun := lastRun.Add(r.config.Operations.Sets[setIdx].Cadence)

if timeutil.Now().Compare(eligibleForNextRun) < 0 {
// Find another operation to run.
r.mu.completed.Wait()
return true
}
```

As seen above, we block on the `completed` `Sync.Cond`. However, the only place we ever `Broadcast` or `Signal` is at the end of `runOperation`. Consider a parallelism of 1, where we select the same operation back to back. Because there are no other operations concurrently running, we would block indefinitely on the `Wait()`. This can also be hit on > 1 parallelism given an unlucky enough scheduling, e.g. all workers are in the `pickOperation` method and select operations that are on cooldown or can't be run concurrently.

Jira issue: CRDB-57132

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.