basalt-26.3: pkg/sql/sqlstats/sqlstats_test: TestDataDrivenTest failed

pkg/sql/sqlstats/sqlstats_test.TestDataDrivenTest [failed](https://mesolite.cluster.engflow.com/invocations/default/376fcf97-274a-4e7c-ad0a-4aafab7d24f7?testReportRun=3&testReportShard=1&testReportAttempt=1#targets-Ly9wa2cvc3FsL3NxbHN0YXRzOnNxbHN0YXRzX3Rlc3Q=) on basalt-26.3 @ [c6d01ae4be9708178b248a49819b505daf713377](https://github.com/cockroachdb/cockroach/commits/c6d01ae4be9708178b248a49819b505daf713377):

```
test_server_shim.go:168: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
test_server_shim.go:639: DRPC is enabled (via testserver factory defaults)
datadriven.go:357: /mnt/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/udf:1: still running after 10.026180945s
sqlstats_test.go:53:
/mnt/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/udf:1:
exec-sql db=udf_test app-name=udf
CREATE TABLE IF NOT EXISTS test(a int);
CREATE FUNCTION random_int()
RETURNS INT
LANGUAGE SQL
AS $$
SELECT floor(random() * 100)::INT
$$
VOLATILE;
CREATE FUNCTION do_something()
RETURNS INT
LANGUAGE SQL
AS $$
INSERT INTO test(select random_int());
SELECT max(a) FROM test;
$$
VOLATILE;
----
datadriven.go:357: /mnt/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/udf:20: still running after 10.002500128s
sqlstats_test.go:53:
/mnt/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/udf:20:
exec-sql db=udf_test app-name=udf
SELECT random_int();
select do_something();
----
sqlstats_test.go:53:
/mnt/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/sqlstats/sqlstats_test_/sqlstats_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/sqlstats/testdata/udf:25:
show-stats db=udf_test app-name=udf
output didn't match expected:
@@ -1,9 +1,9 @@
-{"count": "1", "fingerprint_id": "34c73026fa901192", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": true, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT max(a) FROM udf_test.public.test", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT max(a) FROM udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
-{"count": "1", "fingerprint_id": "5bea038ef190748b", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT random_int()", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT random_int()", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
+{"count": "3", "fingerprint_id": "34c73026fa901192", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": true, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT max(a) FROM udf_test.public.test", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT max(a) FROM udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
+{"count": "3", "fingerprint_id": "5bea038ef190748b", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT random_int()", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT random_int()", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
{"count": "1", "fingerprint_id": "790abc99984823ea", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE FUNCTION random_int()\n\tRETURNS INT8\n\tLANGUAGE SQL\n\tVOLATILE\n\tAS $$_$$", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE FUNCTION random_int()\n\tRETURNS INT8\n\tLANGUAGE SQL\n\tVOLATILE\n\tAS $$_$$", "svc_lat_not_zero": true, "transaction_fingerprint_id": "d66901d51e499435"}
{"count": "1", "fingerprint_id": "82b6e892da6f1a4b", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE FUNCTION do_something()\n\tRETURNS INT8\n\tLANGUAGE SQL\n\tVOLATILE\n\tAS $$_$$", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE FUNCTION do_something()\n\tRETURNS INT8\n\tLANGUAGE SQL\n\tVOLATILE\n\tAS $$_$$", "svc_lat_not_zero": true, "transaction_fingerprint_id": "2dd555de5c6ead94"}
-{"count": "1", "fingerprint_id": "a48f10bb3110825e", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT do_something()", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT do_something()", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
-{"count": "2", "fingerprint_id": "ad8c3186c0ae14a2", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT floor(random() * _)::INT8", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT floor(random() ...)...", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
-{"count": "1", "fingerprint_id": "bf73cd8472f898fd", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "INSERT INTO udf_test.public.test (SELECT public.random_int())", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "INSERT INTO udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
+{"count": "3", "fingerprint_id": "a48f10bb3110825e", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT do_something()", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT do_something()", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
+{"count": "6", "fingerprint_id": "ad8c3186c0ae14a2", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "SELECT floor(random() * _)::INT8", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "SELECT floor(random() ...)...", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
+{"count": "3", "fingerprint_id": "bf73cd8472f898fd", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "INSERT INTO udf_test.public.test (SELECT public.random_int())", "run_lat_not_zero": true, "sql_type": "TypeDML", "summary": "INSERT INTO udf_test.public.test", "svc_lat_not_zero": true, "transaction_fingerprint_id": "b35d68ca1dbe65e2"}
{"count": "1", "fingerprint_id": "f17c354bb29f8432", "nodes": [1], "parse_lat_not_zero": true, "plan_distributed": false, "plan_full_scan": false, "plan_hash_set": true, "plan_lat_not_zero": true, "plan_vectorized": true, "query": "CREATE TABLE IF NOT EXISTS test (a INT8)", "run_lat_not_zero": true, "sql_type": "TypeDDL", "summary": "CREATE TABLE IF NOT EXISTS test (a INT8)", "svc_lat_not_zero": true, "transaction_fingerprint_id": "5e1f8807349e33ed"}

--- FAIL: TestDataDrivenTest/udf (62.04s)
```

Parameters:
- attempt=1
- race=true
- run=3
- shard=1
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachlabs/obs-prs

[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*TestDataDrivenTest.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68094

Start with pkg/sql/sqlstats/sqlstats_test.go and the TestDataDrivenTest case, then inspect pkg/sql/sqlstats/testdata/udf for the failing data-driven steps and expected statistics. Re-run the udf case, compare the timeout and count differences, and consider the issue done when the test completes reliably with matching output.

Written by the indexing model from the issue text.