pkg/sql/sqlstats/sqlstats_test: TestDataDrivenTest failed

pkg/sql/sqlstats/sqlstats_test.TestDataDrivenTest [failed](https://mesolite.cluster.engflow.com/invocations/default/0f992eae-deef-40b2-96b7-6805923c16cd?testReportRun=1&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvc3FsL3NxbHN0YXRzOnNxbHN0YXRzX3Rlc3Q=) on master @ [a2ee11ece63c60d3f50c2c8b12b978cd2e9d1ab5](https://github.com/cockroachdb/cockroach/commits/a2ee11ece63c60d3f50c2c8b12b978cd2e9d1ab5):

```
=== RUN TestDataDrivenTest/plpgsql
test_server_shim.go:168: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:1: still running after 10.013820785s
sqlstats_test.go:53:
/mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:1:
exec-sql db=udf_test app-name=plpgsql
CREATE TABLE IF NOT EXISTS test(a int);
CREATE OR REPLACE PROCEDURE insert_incremental(n INT)
LANGUAGE plpgsql
AS $$
DECLARE
i INT := 0;
next_val INT;
BEGIN
WHILE i < n LOOP
SELECT coalesce(max(a), 0) + 1 INTO next_val FROM test;
INSERT INTO test(a) VALUES (next_val);
i := i + 1;
END LOOP;
END;
$$;
----
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 10.020463855s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 20.024861372s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 30.033411638s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 40.035062085s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 50.057931641s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 1m0.061024479s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 1m10.066050977s
datadriven.go:357: /mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19: still running after 1m20.067171726s
sqlstats_test.go:53:
/mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:19:
exec-sql db=udf_test app-name=plpgsql
CALL insert_incremental(5);
----
sqlstats_test.go:53:
/mnt/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/plpgsql:23:
show-stats db=udf_test app-name=plpgsql
output didn't match expected:
@@ -1,6 +1,6 @@
{"count": "1", "dist_sql_count": 0, "fingerprint_id": "4057632793bd4769", "full_scan_count": 0, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE OR REPLACE PROCEDURE insert_incremental(n INT8)\n\tLANGUAGE plpgsql\n\tAS $$_$$", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE OR REPLACE PROCEDURE insert_incremental(n INT8)\n\tLANGUAGE plpgsql\n\tAS $$_$$", "svc_lat_not_zero": true, "transaction_fingerprint_id": "ef34de6b15bcf0b6", "vec_count": 1}
-{"count": "1", "dist_sql_count": 0, "fingerprint_id": "8c317651ff701c49", "full_scan_count": 0, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CALL insert_incremental(_)", "run_lat_not_zero": true, "sql_type": "TypeTCL", "summary": "CALL insert_incremental(_)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2352cb1d7971ab96", "vec_count": 1}
-{"count": "5", "dist_sql_count": 0, "fingerprint_id": "a457fd0c13d1514f", "full_scan_count": 5, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": true, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT COALESCE(max(a), _) + _ FROM udf_test.public.test", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT COALESCE(max(a)...)... FROM udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2352cb1d7971ab96", "vec_count": 5}
+{"count": "2", "dist_sql_count": 0, "fingerprint_id": "8c317651ff701c49", "full_scan_count": 0, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CALL insert_incremental(_)", "run_lat_not_zero": true, "sql_type": "TypeTCL", "summary": "CALL insert_incremental(_)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2352cb1d7971ab96", "vec_count": 2}
+{"count": "10", "dist_sql_count": 0, "fingerprint_id": "a457fd0c13d1514f", "full_scan_count": 10, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": true, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT COALESCE(max(a), _) + _ FROM udf_test.public.test", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT COALESCE(max(a)...)... FROM udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2352cb1d7971ab96", "vec_count": 10}
{"count": "1", "dist_sql_count": 0, "fingerprint_id": "f17c354bb29f8432", "full_scan_count": 0, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE TABLE IF NOT EXISTS test (a INT8)", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE TABLE IF NOT EXISTS test (a INT8)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "5e1f8807349e33ed", "vec_count": 1}
-{"count": "5", "dist_sql_count": 0, "fingerprint_id": "f71d046ecfc2a04f", "full_scan_count": 0, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "INSERT INTO udf_test.public.test(a) VALUES (next_val)", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "INSERT INTO udf_test.public.test(a)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2352cb1d7971ab96", "vec_count": 5}
+{"count": "10", "dist_sql_count": 0, "fingerprint_id": "f71d046ecfc2a04f", "full_scan_count": 0, "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "INSERT INTO udf_test.public.test(a) VALUES (next_val)", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "INSERT INTO udf_test.public.test(a)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2352cb1d7971ab96", "vec_count": 10}

--- FAIL: TestDataDrivenTest/plpgsql (128.60s)
```

Parameters:
- attempt=1
- race=true
- run=1
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #173936 basalt-26.3: pkg/sql/sqlstats/sqlstats_test: TestDataDrivenTest failed [C-test-failure O-robot T-observability branch-basalt-26.3 release-blocker]
- #173235 release-26.3.0-rc: pkg/sql/sqlstats/sqlstats_test: TestDataDrivenTest failed [C-test-failure O-robot T-observability branch-release-26.3.0-rc]

/cc @cockroachlabs/obs-prs

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-67641

Start with pkg/sql/sqlstats/sqlstats_test.go and the TestDataDrivenTest case, then inspect pkg/sql/sqlstats/testdata/plpgsql and the reported failure output. Reproduce the plpgsql case and determine why the procedure's SQL statistics remain running and produce doubled counts; done means the test completes and its expected output matches consistently.

Written by the indexing model from the issue text.