bench/rttanalysis: TestBenchmarkExpectationShard1 failed [crdb_internal.system_jobs reads]

bench/rttanalysis.TestBenchmarkExpectationShard1 [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/20549007?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/20549007?buildTab=artifacts#/) on master @ [552cfcb086721820f921ed04c08af7e2df6eabcf](https://github.com/cockroachdb/cockroach/commits/552cfcb086721820f921ed04c08af7e2df6eabcf):

```
I251002 17:15:45.887876 1 (gostd) rand.go:268 [-] 1 random seed: 1191610951310289352
validate_benchmark_data.go:186: [/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/10619/execroot/com_github_cockroachdb_cockroach/bazel-out/s390x-fastbuild/bin/pkg/bench/rttanalysis/rttanalysis_test_/rttanalysis_test.runfiles/com_github_cockroachdb_cockroach/pkg/bench/rttanalysis/rttanalysis_test_/rttanalysis_test --test.v=test2json --test.bench=^$]
I251002 17:15:46.458841 1 (gostd) rand.go:268 [-] 1 random seed: 2154924018993046327
=== RUN TestBenchmarkExpectationShard1
test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/3e7abc59ef4f9b4e3bad6013583e698d/logTestBenchmarkExpectationShard11028806542
test_log_scope.go:82: use -show-logs to present logs inline
validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_3_values to perform KV lookups in [14, 14], got 14
validate_benchmark_data.go:120: success: expected UDFResolution/select_from_udf to perform KV lookups in [0, 0], got 0
validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_2_values to perform KV lookups in [11, 11], got 11
validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_3_options to perform KV lookups in [19, 19], got 19
validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_5_replicas to perform KV lookups in [14, 14], got 14
validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_7_replicas_ to perform KV lookups in [14, 14], got 14
validate_benchmark_data.go:120: success: expected DropView/drop_3_views to perform KV lookups in [15, 15], got 15
validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_ranges to perform KV lookups in [14, 14], got 14
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_3_tables to perform KV lookups in [16, 16], got 16
validate_benchmark_data.go:120: success: expected ShowGrants/grant_2_roles to perform KV lookups in [4, 4], got 4
validate_benchmark_data.go:120: success: expected ShowGrants/grant_3_roles to perform KV lookups in [4, 4], got 4
validate_benchmark_data.go:120: success: expected ShowGrants/grant_4_roles to perform KV lookups in [4, 4], got 4
validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_1_value to perform KV lookups in [8, 8], got 8
validate_benchmark_data.go:120: success: expected DropView/drop_1_view to perform KV lookups in [14, 14], got 14
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_0_tables to perform KV lookups in [15, 15], got 15
validate_benchmark_data.go:120: success: expected DropView/drop_2_views to perform KV lookups in [15, 15], got 15
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_1_table to perform KV lookups in [16, 16], got 16
validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_1_option to perform KV lookups in [12, 12], got 12
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_type_filtered_no_matches to perform KV lookups in [1, 3], got 1
validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_2_tables to perform KV lookups in [16, 16], got 16
validate_benchmark_data.go:120: success: expected Jobs/show_job to perform KV lookups in [3, 3], got 3
validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_2_options to perform KV lookups in [15, 15], got 15
validate_benchmark_data.go:120: success: expected Jobs/show_jobs to perform KV lookups in [3, 5], got 3
validate_benchmark_data.go:120: success: expected Jobs/pause_job to perform KV lookups in [12, 15], got 15
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_type_filtered to perform KV lookups in [3, 3], got 3
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_latest_50 to perform KV lookups in [3, 3], got 3
validate_benchmark_data.go:120: success: expected Jobs/resume_job to perform KV lookups in [12, 15], got 13
validate_benchmark_data.go:120: success: expected Jobs/cancel_job to perform KV lookups in [12, 15], got 13
validate_benchmark_data.go:120: success: expected Jobs/jobs_page_default to perform KV lookups in [3, 5], got 3
validate_benchmark_data.go:117: fail: expected Jobs/crdb_internal.system_jobs to perform KV lookups in [3, 3], got 10
validate_benchmark_data.go:120: success: expected Jobs/non_admin_show_jobs to perform KV lookups in [2, 2], got 2
validate_benchmark_data.go:120: success: expected Jobs/non_admin_crdb_internal.system_jobs to perform KV lookups in [2, 2], got 2
validate_benchmark_data.go:73: see the -rewrite flag to re-run the benchmarks and adjust the expectations
validate_benchmark_data.go:74: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
validate_benchmark_data.go:107: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/3e7abc59ef4f9b4e3bad6013583e698d/logTestBenchmarkExpectationShard11028806542
--- FAIL: TestBenchmarkExpectationShard1 (311.18s)
--- FAIL: TestBenchmarkExpectationShard1 (312.36s)
=== RUN TestBenchmarkExpectationShard1/Jobs
--- FAIL: TestBenchmarkExpectationShard1/Jobs (311.16s)
=== RUN TestBenchmarkExpectationShard1/Jobs/crdb_internal.system_jobs
rtt_analysis_bench.go:224: Jobs/crdb_internal.system_jobs: got 10, expected 3
rtt_analysis_bench.go:230: wrote jaeger trace to /artifacts/tmp/_tmp/3e7abc59ef4f9b4e3bad6013583e698d/logTestBenchmarkExpectationShard11028806542/Jobs_crdb_internal.system_jobs.jaeger.json
--- FAIL: TestBenchmarkExpectationShard1/Jobs/crdb_internal.system_jobs (291.26s)
```
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/sql-foundations

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestBenchmarkExpectationShard1.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-55067

Start in pkg/bench/rttanalysis with TestBenchmarkExpectationShard1 and the Jobs/crdb_internal.system_jobs case; read validate_benchmark_data.go and rtt_analysis_bench.go, then rerun the benchmark using the command shown in the failure. Use the emitted Jaeger trace and logs to determine why 10 KV lookups occur instead of 3, and finish with the benchmark expectation passing.

Written by the indexing model from the issue text.