bench/rttanalysis: TestBenchmarkExpectationShard1 failed

bench/rttanalysis.TestBenchmarkExpectationShard1 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/20988362?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsAwsLinuxArm64_UnitTests/20988362?buildTab=artifacts#/) on release-26.1 @ [493a33b75bad03f11ae646a5191737d013d57634](https://github.com/cockroachdb/cockroach/commits/493a33b75bad03f11ae646a5191737d013d57634):

```
=== RUN TestBenchmarkExpectationShard1
validate_benchmark_data.go:186: [/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/19314/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/bench/rttanalysis/rttanalysis_test_/rttanalysis_test.runfiles/com_github_cockroachdb_cockroach/pkg/bench/rttanalysis/rttanalysis_test_/rttanalysis_test --test.v=test2json --test.bench=^$]
I260116 18:55:18.696312 1 (gostd) rand.go:288 [-] 1 random seed: 6167433681384202704
=== RUN TestBenchmarkExpectationShard1
test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectationShard11642089614
test_log_scope.go:82: use -show-logs to present logs inline
validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_1_value to perform KV lookups in [8, 8], got 8
validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_2_options to perform KV lookups in [15, 15], got 16
validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_3_values to perform KV lookups in [14, 14], got 14
validate_benchmark_data.go:120: success: expected UDFResolution/select_from_udf to perform KV lookups in [0, 0], got 0
validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_2_values to perform KV lookups in [11, 11], got 11
validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_1_option to perform KV lookups in [12, 12], got 13
validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_3_options to perform KV lookups in [19, 19], got 20
validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_5_replicas to perform KV lookups in [14, 14], got 15
validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_7_replicas_ to perform KV lookups in [14, 14], got 15
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_0_tables to perform KV lookups in [15, 15], got 16
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_3_tables to perform KV lookups in [16, 16], got 17
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_2_tables to perform KV lookups in [16, 16], got 17
validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_ranges to perform KV lookups in [14, 14], got 15
validate_benchmark_data.go:120: success: expected DropView/drop_1_view to perform KV lookups in [14, 14], got 15
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_1_table to perform KV lookups in [16, 16], got 17
validate_benchmark_data.go:120: success: expected DropView/drop_2_views to perform KV lookups in [15, 15], got 16
validate_benchmark_data.go:120: success: expected DropView/drop_3_views to perform KV lookups in [15, 15], got 16
validate_benchmark_data.go:120: success: expected ShowGrants/grant_2_roles to perform KV lookups in [4, 4], got 4
validate_benchmark_data.go:120: success: expected ShowGrants/grant_3_roles to perform KV lookups in [4, 4], got 4
validate_benchmark_data.go:120: success: expected ShowGrants/grant_4_roles to perform KV lookups in [4, 4], got 4
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_type_filtered_no_matches to perform KV lookups in [1, 3], got 1
validate_benchmark_data.go:120: success: expected Jobs/show_job to perform KV lookups in [3, 3], got 3
validate_benchmark_data.go:120: success: expected Jobs/pause_job to perform KV lookups in [9, 12], got 10
validate_benchmark_data.go:120: success: expected Jobs/cancel_job to perform KV lookups in [9, 12], got 9
validate_benchmark_data.go:120: success: expected Jobs/resume_job to perform KV lookups in [9, 12], got 9
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_latest_50 to perform KV lookups in [3, 3], got 3
validate_benchmark_data.go:120: success: expected Jobs/show_jobs to perform KV lookups in [3, 5], got 3
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_type_filtered to perform KV lookups in [3, 3], got 3
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_default to perform KV lookups in [3, 5], got 3
validate_benchmark_data.go:117: fail: expected Jobs/crdb_internal.system_jobs to perform KV lookups in [3, 3], got 10
validate_benchmark_data.go:120: success: expected Jobs/non_admin_show_jobs to perform KV lookups in [2, 2], got 2
validate_benchmark_data.go:120: success: expected Jobs/non_admin_crdb_internal.system_jobs to perform KV lookups in [2, 2], got 2
validate_benchmark_data.go:73: see the -rewrite flag to re-run the benchmarks and adjust the expectations
validate_benchmark_data.go:74: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
validate_benchmark_data.go:107: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectationShard11642089614
--- FAIL: TestBenchmarkExpectationShard1 (140.99s)
--- FAIL: TestBenchmarkExpectationShard1 (141.36s)
=== RUN TestBenchmarkExpectationShard1/Jobs/crdb_internal.system_jobs
rtt_analysis_bench.go:224: Jobs/crdb_internal.system_jobs: got 10, expected 3
rtt_analysis_bench.go:230: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectationShard11642089614/Jobs_crdb_internal.system_jobs.jaeger.json
--- FAIL: TestBenchmarkExpectationShard1/Jobs/crdb_internal.system_jobs (105.96s)
=== RUN TestBenchmarkExpectationShard1/Jobs
--- FAIL: TestBenchmarkExpectationShard1/Jobs (140.98s)
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #154668 bench/rttanalysis: TestBenchmarkExpectationShard1 failed [crdb_internal.system_jobs reads] [A-jobs C-test-failure O-robot P-3 T-jobs branch-master release-blocker s390x-test-failure]

/cc @cockroachdb/sql-foundations

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestBenchmarkExpectationShard1.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-58776