cockroachdb / cockroachdb/cockroach
release-26.3.0-rc: server/application_api: TestStatusAPICombinedStatementsWithFullScans failed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
server/application_api.TestStatusAPICombinedStatementsWithFullScans [failed](https://mesolite.cluster.engflow.com/invocations/default/218a8d52-e21b-4124-8282-cb673e244dfd?testReportRun=3&testReportShard=2&testReportAttempt=1#targets-Ly9wa2cvc2VydmVyL2FwcGxpY2F0aW9uX2FwaTphcHBsaWNhdGlvbl9hcGlfdGVzdA==) on release-26.3.0-rc @ [0a3dc2524e335b85175c507a318dd57f28f92ef3](https://github.com/cockroachdb/cockroach/commits/0a3dc2524e335b85175c507a318dd57f28f92ef3):
```
=== RUN TestStatusAPICombinedStatementsWithFullScans
test_log_scope.go:172: test logs captured to: outputs.zip/logTestStatusAPICombinedStatementsWithFullScans3756074942
test_log_scope.go:82: use -show-logs to present logs inline
sql_stats_test.go:854: condition failed to evaluate within 45s: expected statement 'INSERT INTO players(id, name, "position", age, goals) VALUES (_, __more__), (__more__)' not found in response: {"statements":[{"key":{"key_data":{"query":"GRANT SYSTEM VIEWACTIVITY TO authentic_user_noadmin","app":"","distSQL":false,"vec":false,"full_scan":false,"database":"defaultdb","plan_hash":0,"query_summary":"GRANT SYSTEM VIEWACTIVITY TO authentic_user_noadmin","transaction_fingerprint_id":0},"aggregated_ts":"2026-07-30T05:00:00Z","aggregation_interval":0},"id":870064593363012594,"stats":{"count":1,"first_attempt_count":1,"max_retries":0,"legacy_last_err":"","legacy_last_err_redacted":"","num_rows":{"mean":0,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"parse_lat":{"mean":0.000018239,"squared_diffs":0},"plan_lat":{"mean":0.000306234,"squared_diffs":0},"run_lat":{"mean":0.617409026,"squared_diffs":0},"service_lat":{"mean":0.617736569,"squared_diffs":0},"overhead_lat":{"mean":0.00000307,"squared_diffs":0},"sensitive_info":{"last_err":"","most_recent_plan_description":{"name":""},"most_recent_plan_timestamp":"0001-01-01T00:00:00Z"},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"rows_written":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":2261816,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":2961,"squared_diffs":0}},"sql_type":"TypeDCL","last_exec_timestamp":"2026-07-30T05:59:59.971830679Z","nodes":[3],"regions":["test"],"plan_gists":["Ais="],"latency_info":{"min":0.617736569,"max":0.617736569,"p50":0,"p90":0,"p99":0},"last_error_code":"","failure_count":0,"used_follower_read":false,"generic_count":0,"stmt_hints_count":0,"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0},"canary_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}},"stable_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}}},"txn_fingerprint_ids":[0]},{"key":{"key_data":{"query":"SET application_name = _","app":"","distSQL":false,"vec":false,"full_scan":false,"database":"defaultdb","plan_hash":0,"query_summary":"SET application_name = _","transaction_fingerprint_id":0},"aggregated_ts":"2026-07-30T05:00:00Z","aggregation_interval":0},"id":12121160597728358152,"stats":{"count":1,"first_attempt_count":1,"max_retries":0,"legacy_last_err":"","legacy_last_err_redacted":"","num_rows":{"mean":0,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"parse_lat":{"mean":0.00001533,"squared_diffs":0},"plan_lat":{"mean":0.000068758,"squared_diffs":0},"run_lat":{"mean":0.000145477,"squared_diffs":0},"service_lat":{"mean":0.000231695,"squared_diffs":0},"overhead_lat":{"mean":0.00000213,"squared_diffs":0},"sensitive_info":{"last_err":"","most_recent_plan_description":{"name":""},"most_recent_plan_timestamp":"0001-01-01T00:00:00Z"},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"rows_written":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":215526,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":0,"squared_diffs":0}},"sql_type":"TypeDML","last_exec_timestamp":"2026-07-30T05:59:59.972491815Z","nodes":[3],"regions":["test"],"plan_gists":["Ais="],"latency_info":{"min":0.000231695,"max":0.000231695,"p50":0,"p90":0,"p99":0},"last_error_code":"","failure_count":0,"used_follower_read":false,"generic_count":0,"stmt_hints_count":0,"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0},"canary_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}},"stable_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}}},"txn_fingerprint_ids":[0]}],"last_reset":"2026-07-30T05:59:54.006706646Z","internal_app_name_prefix":"$ internal","transactions":[{"stats_data":{"statement_fingerprint_ids":[12121160597728358152],"app":"","stats":{"count":1,"max_retries":0,"num_rows":{"mean":0,"squared_diffs":0},"service_lat":{"mean":0.000345963,"squared_diffs":0},"retry_lat":{"mean":0,"squared_diffs":0},"commit_lat":{"mean":0.000007429,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":215526,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":0,"squared_diffs":0}},"rows_written":{"mean":0,"squared_diffs":0},"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0}},"aggregated_ts":"2026-07-30T05:00:00Z","transaction_fingerprint_id":528256080741849303,"aggregation_interval":0}},{"stats_data":{"statement_fingerprint_ids":[870064593363012594],"app":"","stats":{"count":1,"max_retries":0,"num_rows":{"mean":0,"squared_diffs":0},"service_lat":{"mean":1.223749489,"squared_diffs":0},"retry_lat":{"mean":0,"squared_diffs":0},"commit_lat":{"mean":0.062882145,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":2261816,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":2961,"squared_diffs":0}},"rows_written":{"mean":0,"squared_diffs":0},"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0}},"aggregated_ts":"2026-07-30T05:00:00Z","transaction_fingerprint_id":11777100422732168237,"aggregation_interval":0}}],"stmts_total_runtime_secs":3.7455232,"txns_total_runtime_secs":3.2624855,"oldest_aggregated_ts_returned":"2026-07-30T05:00:00Z","stmts_source_table":"crdb_internal.statement_statistics","txns_source_table":"crdb_internal.transaction_statistics"}
goroutine dump: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/application_api/application_api_test/shard_2_of_8_run_3_of_10/test.outputs/goroutine_dump_3263931047.txt
panic.go:694: -- test log scope end --
test logs left over in: outputs.zip/logTestStatusAPICombinedStatementsWithFullScans3756074942
--- FAIL: TestStatusAPICombinedStatementsWithFullScans (53.85s)
```
Parameters:
- attempt=1
- run=3
- shard=2
Help
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
/cc @cockroachlabs/obs-prs
[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestStatusAPICombinedStatementsWithFullScans.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
Jira issue: CRDB-66235
Contributor guide
Research direction
Start with pkg/server/application_api/sql_stats_test.go around line 854 and run TestStatusAPICombinedStatementsWithFullScans using the linked EngFlow invocation and captured logs. Determine why the expected INSERT statement is absent from the Status API response; done means the test reliably finds the expected statement and passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- api, databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100