cockroachdb / cockroachdb/cockroach

release-26.3.0-rc: server/application_api: TestStatusAPICombinedStatementsWithFullScans failed

Open
#172,946 2 comments 0 reactions 0 assignees View on GitHub
branch-release-26.3.0-rc C-test-failure O-robot T-observability v26.3.1
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

server/application_api.TestStatusAPICombinedStatementsWithFullScans [failed](https://mesolite.cluster.engflow.com/invocations/default/218a8d52-e21b-4124-8282-cb673e244dfd?testReportRun=3&testReportShard=2&testReportAttempt=1#targets-Ly9wa2cvc2VydmVyL2FwcGxpY2F0aW9uX2FwaTphcHBsaWNhdGlvbl9hcGlfdGVzdA==) on release-26.3.0-rc @ [0a3dc2524e335b85175c507a318dd57f28f92ef3](https://github.com/cockroachdb/cockroach/commits/0a3dc2524e335b85175c507a318dd57f28f92ef3):

```
=== RUN TestStatusAPICombinedStatementsWithFullScans
test_log_scope.go:172: test logs captured to: outputs.zip/logTestStatusAPICombinedStatementsWithFullScans3756074942
test_log_scope.go:82: use -show-logs to present logs inline
sql_stats_test.go:854: condition failed to evaluate within 45s: expected statement 'INSERT INTO players(id, name, "position", age, goals) VALUES (_, __more__), (__more__)' not found in response: {"statements":[{"key":{"key_data":{"query":"GRANT SYSTEM VIEWACTIVITY TO authentic_user_noadmin","app":"","distSQL":false,"vec":false,"full_scan":false,"database":"defaultdb","plan_hash":0,"query_summary":"GRANT SYSTEM VIEWACTIVITY TO authentic_user_noadmin","transaction_fingerprint_id":0},"aggregated_ts":"2026-07-30T05:00:00Z","aggregation_interval":0},"id":870064593363012594,"stats":{"count":1,"first_attempt_count":1,"max_retries":0,"legacy_last_err":"","legacy_last_err_redacted":"","num_rows":{"mean":0,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"parse_lat":{"mean":0.000018239,"squared_diffs":0},"plan_lat":{"mean":0.000306234,"squared_diffs":0},"run_lat":{"mean":0.617409026,"squared_diffs":0},"service_lat":{"mean":0.617736569,"squared_diffs":0},"overhead_lat":{"mean":0.00000307,"squared_diffs":0},"sensitive_info":{"last_err":"","most_recent_plan_description":{"name":""},"most_recent_plan_timestamp":"0001-01-01T00:00:00Z"},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"rows_written":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":2261816,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":2961,"squared_diffs":0}},"sql_type":"TypeDCL","last_exec_timestamp":"2026-07-30T05:59:59.971830679Z","nodes":[3],"regions":["test"],"plan_gists":["Ais="],"latency_info":{"min":0.617736569,"max":0.617736569,"p50":0,"p90":0,"p99":0},"last_error_code":"","failure_count":0,"used_follower_read":false,"generic_count":0,"stmt_hints_count":0,"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0},"canary_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}},"stable_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}}},"txn_fingerprint_ids":[0]},{"key":{"key_data":{"query":"SET application_name = _","app":"","distSQL":false,"vec":false,"full_scan":false,"database":"defaultdb","plan_hash":0,"query_summary":"SET application_name = _","transaction_fingerprint_id":0},"aggregated_ts":"2026-07-30T05:00:00Z","aggregation_interval":0},"id":12121160597728358152,"stats":{"count":1,"first_attempt_count":1,"max_retries":0,"legacy_last_err":"","legacy_last_err_redacted":"","num_rows":{"mean":0,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"parse_lat":{"mean":0.00001533,"squared_diffs":0},"plan_lat":{"mean":0.000068758,"squared_diffs":0},"run_lat":{"mean":0.000145477,"squared_diffs":0},"service_lat":{"mean":0.000231695,"squared_diffs":0},"overhead_lat":{"mean":0.00000213,"squared_diffs":0},"sensitive_info":{"last_err":"","most_recent_plan_description":{"name":""},"most_recent_plan_timestamp":"0001-01-01T00:00:00Z"},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"rows_written":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":215526,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":0,"squared_diffs":0}},"sql_type":"TypeDML","last_exec_timestamp":"2026-07-30T05:59:59.972491815Z","nodes":[3],"regions":["test"],"plan_gists":["Ais="],"latency_info":{"min":0.000231695,"max":0.000231695,"p50":0,"p90":0,"p99":0},"last_error_code":"","failure_count":0,"used_follower_read":false,"generic_count":0,"stmt_hints_count":0,"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0},"canary_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}},"stable_stats":{"count":0,"run_lat":{"mean":0,"squared_diffs":0},"plan_lat":{"mean":0,"squared_diffs":0}}},"txn_fingerprint_ids":[0]}],"last_reset":"2026-07-30T05:59:54.006706646Z","internal_app_name_prefix":"$ internal","transactions":[{"stats_data":{"statement_fingerprint_ids":[12121160597728358152],"app":"","stats":{"count":1,"max_retries":0,"num_rows":{"mean":0,"squared_diffs":0},"service_lat":{"mean":0.000345963,"squared_diffs":0},"retry_lat":{"mean":0,"squared_diffs":0},"commit_lat":{"mean":0.000007429,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":215526,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":0,"squared_diffs":0}},"rows_written":{"mean":0,"squared_diffs":0},"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0}},"aggregated_ts":"2026-07-30T05:00:00Z","transaction_fingerprint_id":528256080741849303,"aggregation_interval":0}},{"stats_data":{"statement_fingerprint_ids":[870064593363012594],"app":"","stats":{"count":1,"max_retries":0,"num_rows":{"mean":0,"squared_diffs":0},"service_lat":{"mean":1.223749489,"squared_diffs":0},"retry_lat":{"mean":0,"squared_diffs":0},"commit_lat":{"mean":0.062882145,"squared_diffs":0},"idle_lat":{"mean":0,"squared_diffs":0},"bytes_read":{"mean":0,"squared_diffs":0},"rows_read":{"mean":0,"squared_diffs":0},"exec_stats":{"count":1,"network_bytes":{"mean":0,"squared_diffs":0},"max_mem_usage":{"mean":10240,"squared_diffs":0},"contention_time":{"mean":0,"squared_diffs":0},"network_messages":{"mean":0,"squared_diffs":0},"max_disk_usage":{"mean":0,"squared_diffs":0},"cpu_sql_nanos":{"mean":2261816,"squared_diffs":0},"mvcc_iterator_stats":{"step_count":{"mean":0,"squared_diffs":0},"step_count_internal":{"mean":0,"squared_diffs":0},"seek_count":{"mean":0,"squared_diffs":0},"seek_count_internal":{"mean":0,"squared_diffs":0},"block_bytes":{"mean":0,"squared_diffs":0},"block_bytes_in_cache":{"mean":0,"squared_diffs":0},"key_bytes":{"mean":0,"squared_diffs":0},"value_bytes":{"mean":0,"squared_diffs":0},"point_count":{"mean":0,"squared_diffs":0},"points_covered_by_range_tombstones":{"mean":0,"squared_diffs":0},"range_key_count":{"mean":0,"squared_diffs":0},"range_key_contained_points":{"mean":0,"squared_diffs":0},"range_key_skipped_points":{"mean":0,"squared_diffs":0}},"admission_wait_time":{"mean":2961,"squared_diffs":0}},"rows_written":{"mean":0,"squared_diffs":0},"kv_cpu_time_nanos":{"mean":0,"squared_diffs":0}},"aggregated_ts":"2026-07-30T05:00:00Z","transaction_fingerprint_id":11777100422732168237,"aggregation_interval":0}}],"stmts_total_runtime_secs":3.7455232,"txns_total_runtime_secs":3.2624855,"oldest_aggregated_ts_returned":"2026-07-30T05:00:00Z","stmts_source_table":"crdb_internal.statement_statistics","txns_source_table":"crdb_internal.transaction_statistics"}

goroutine dump: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/server/application_api/application_api_test/shard_2_of_8_run_3_of_10/test.outputs/goroutine_dump_3263931047.txt
panic.go:694: -- test log scope end --
test logs left over in: outputs.zip/logTestStatusAPICombinedStatementsWithFullScans3756074942
--- FAIL: TestStatusAPICombinedStatementsWithFullScans (53.85s)
```

Parameters:
- attempt=1
- run=3
- shard=2
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/obs-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestStatusAPICombinedStatementsWithFullScans.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66235

Contributor guide

Open the contributing guide

Research direction

Start with pkg/server/application_api/sql_stats_test.go around line 854 and run TestStatusAPICombinedStatementsWithFullScans using the linked EngFlow invocation and captured logs. Determine why the expected INSERT statement is absent from the Status API response; done means the test reliably finds the expected statement and passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
api, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.