release-26.3.0-rc: pkg/sql/sqlstats/sqlstats_test: TestDataDrivenTest failed

pkg/sql/sqlstats/sqlstats_test.TestDataDrivenTest [failed](https://mesolite.cluster.engflow.com/invocations/default/5c02ab6a-3c30-4b0b-b810-51a5cf7c710e?testReportRun=3&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvc3FsL3NxbHN0YXRzOnNxbHN0YXRzX3Rlc3Q=) on release-26.3.0-rc @ [0a3dc2524e335b85175c507a318dd57f28f92ef3](https://github.com/cockroachdb/cockroach/commits/0a3dc2524e335b85175c507a318dd57f28f92ef3):

```
=== RUN TestDataDrivenTest
test_log_scope.go:172: test logs captured to: outputs.zip/logTestDataDrivenTest900572207
test_log_scope.go:82: use -show-logs to present logs inline
sqlstats_test.go:99: -- test log scope end --
test logs left over in: outputs.zip/logTestDataDrivenTest900572207
--- FAIL: TestDataDrivenTest (166.09s)
=== RUN TestDataDrivenTest/plpgsql
test_server_shim.go:168: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
datadriven.go:357: /mnt/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:1: still running after 10.000568458s
datadriven.go:357: /mnt/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:1: still running after 20.066691648s
sqlstats_test.go:53:
/mnt/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:1:
exec-sql db=udf_test app-name=plpgsql
CREATE TABLE IF NOT EXISTS test(a int);
CREATE OR REPLACE PROCEDURE insert_incremental(n INT)
LANGUAGE plpgsql
AS $$
DECLARE
i INT := 0;
next_val INT;
BEGIN
WHILE i < n LOOP
SELECT coalesce(max(a), 0) + 1 INTO next_val FROM test;
INSERT INTO test(a) VALUES (next_val);
i := i + 1;
END LOOP;
END;
$$;
CALL insert_incremental(5);
----
sqlstats_test.go:53:
/mnt/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:20:
show-stats db=udf_test app-name=plpgsql
output didn't match expected:
@@ -1,6 +1,6 @@
-{"count": "1", "fingerprint_id": "4057632793bd4769", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE OR REPLACE PROCEDURE insert_incremental(n INT8)\n\tLANGUAGE plpgsql\n\tAS $$_$$", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE OR REPLACE PROCEDURE insert_incremental(n INT8)\n\tLANGUAGE plpgsql\n\tAS $$_$$", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
-{"count": "1", "fingerprint_id": "8c317651ff701c49", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CALL insert_incremental(_)", "run_lat_not_zero": true, "sql_type": "TypeTCL", "summary": "CALL insert_incremental(_)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
-{"count": "5", "fingerprint_id": "a457fd0c13d1514f", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": true, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT COALESCE(max(a), _) + _ FROM udf_test.public.test", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT COALESCE(max(a)...)... FROM udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
+{"count": "3", "fingerprint_id": "4057632793bd4769", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE OR REPLACE PROCEDURE insert_incremental(n INT8)\n\tLANGUAGE plpgsql\n\tAS $$_$$", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE OR REPLACE PROCEDURE insert_incremental(n INT8)\n\tLANGUAGE plpgsql\n\tAS $$_$$", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
+{"count": "3", "fingerprint_id": "8c317651ff701c49", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CALL insert_incremental(_)", "run_lat_not_zero": true, "sql_type": "TypeTCL", "summary": "CALL insert_incremental(_)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
+{"count": "13", "fingerprint_id": "a457fd0c13d1514f", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": true, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT COALESCE(max(a), _) + _ FROM udf_test.public.test", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT COALESCE(max(a)...)... FROM udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
{"count": "1", "fingerprint_id": "f17c354bb29f8432", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE TABLE IF NOT EXISTS test (a INT8)", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE TABLE IF NOT EXISTS test (a INT8)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "5e1f8807349e33ed"}
-{"count": "5", "fingerprint_id": "f71d046ecfc2a04f", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "INSERT INTO udf_test.public.test(a) VALUES (next_val)", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "INSERT INTO udf_test.public.test(a)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}
+{"count": "12", "fingerprint_id": "f71d046ecfc2a04f", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "INSERT INTO udf_test.public.test(a) VALUES (next_val)", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "INSERT INTO udf_test.public.test(a)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "bff7d3a40f7d190b"}

--- FAIL: TestDataDrivenTest/plpgsql (62.91s)
```

Parameters:
- attempt=1
- race=true
- run=3
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #172821 release-26.3: pkg/sql/sqlstats/sqlstats_test: TestDataDrivenTest failed [C-test-failure O-robot P-2 T-sql-foundations branch-release-26.3]

/cc @cockroachlabs/obs-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestDataDrivenTest.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-66551

Start with pkg/sql/sqlstats/sqlstats_test.go and the pkg/sql/sqlstats/testdata/plpgsql data-driven test, then read the referenced test_server_shim.go and datadriven.go behavior. Reproduce TestDataDrivenTest/plpgsql with the reported release commit and inspect the captured logs. Done means identifying and fixing the cause of the timeout or incorrect SQL statistics, with the test passing consistently.

Written by the indexing model from the issue text.