cockroachdb / cockroachdb/cockroach

sql: make statement diagnostics bundle collection more robust to optimizer errors

Open
#113,300 1 comment 0 reactions 0 assignees View on GitHub
A-sql-debug-bundle A-sql-explain C-bug E-quick-win O-support P-3 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When investigating unexpected internal errors from the optimizer, it would be helpful to have more of the contents of a statement diagnostics bundle, even if we could not produce the plan. Right now we only get the statement, the environment, and the trace, so we are forced to use a bunch of manual SHOW statements to collect schemas and stats.

Here's a demonstration using b5ddb408eefbe550c704abe9ac9838324dfa0c7b with a slight modification:

```
diff --git a/pkg/sql/opt/xform/select_funcs.go b/pkg/sql/opt/xform/select_funcs.go
index c7e37477503..cd7d2abbe57 100644
--- a/pkg/sql/opt/xform/select_funcs.go
+++ b/pkg/sql/opt/xform/select_funcs.go
@@ -24,6 +24,7 @@ import (
"github.com/cockroachdb/cockroach/pkg/sql/sem/tree"
"github.com/cockroachdb/cockroach/pkg/sql/types"
"github.com/cockroachdb/cockroach/pkg/util/intsets"
+ "github.com/cockroachdb/errors"
)

// IsLocking returns true if the ScanPrivate is configured to use a row-level
@@ -915,6 +916,8 @@ func (c *CustomFuncs) GenerateInvertedIndexScans(
sb.AddSelect(filters)

sb.Build(grp)
+
+ panic(errors.AssertionFailedf("oh no!"))
})
}
```

When we try to collect a bundle, we see the error, but can still download the bundle:

```
demo@127.0.0.1:26257/demoapp/defaultdb> CREATE TABLE ij (i INT PRIMARY KEY, j JSONB, INVERTED INDEX (j));
CREATE TABLE

Time: 4ms total (execution 4ms / network 0ms)

demo@127.0.0.1:26257/demoapp/defaultdb> EXPLAIN ANALYZE (DEBUG) SELECT * FROM ij WHERE j->'x' = '1';
ERROR: internal error: oh no!
SQLSTATE: XX000
DETAIL: Statement diagnostics bundle generated. Download using the SQL shell or command
line.
SQL shell: \statement-diag download 912968924944007169
Command line: cockroach statement-diag download 912968924944007169stack trace:
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/select_funcs.go:920: func1()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:304: ForEachStartingAfter()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/scan_index_iter.go:207: ForEach()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/select_funcs.go:834: GenerateInvertedIndexScans()
github.com/cockroachdb/cockroach/bazel-out/darwin-fastbuild/bin/pkg/sql/opt/xform/explorer.og.go:343: exploreSelect()
github.com/cockroachdb/cockroach/bazel-out/darwin-fastbuild/bin/pkg/sql/opt/xform/explorer.og.go:26: exploreGroupMember()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/explorer.go:185: exploreGroup()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:537: optimizeGroup()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:718: optimizeEnforcer()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:669: enforceProps()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:564: optimizeGroupMember()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:524: optimizeGroup()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:270: Optimize()
github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:598: buildExecMemo()
github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:240: makeOptimizerPlan()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2164: makeExecPlan()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1656: dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1071: execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145: func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3256: execWithProfiling()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:144: execStmt()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2227: func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2232: execCmd()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2149: run()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:952: ServeConn()
github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:247: processCommands()
github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:1001: func3()
src/runtime/asm_amd64.s:1650: goexit()

HINT: You have encountered an unexpected error.

Please check the public issue tracker to check whether this problem is
already tracked. If you cannot find it there, please report the error
with details by creating a new issue.

If you would rather not post publicly, please contact us directly
using the support form.

We appreciate your feedback.
demo@127.0.0.1:26257/demoapp/defaultdb> select * from system.statement_diagnostics_requests;
id | completed | statement_fingerprint | statement_diagnostics_id | requested_at | min_execution_latency | expires_at | sampling_probability | plan_gist | anti_plan_gist
---------------------+-----------+---------------------------------------------------------------+--------------------------+-------------------------------+-----------------------+------------+----------------------+-----------+-----------------
912968924947611649 | t | EXPLAIN ANALYZE (DEBUG) SELECT * FROM ij WHERE (j->'_') = '_' | 912968924944007169 | 2023-10-30 17:20:04.924363+00 | NULL | NULL | NULL | NULL | NULL
(1 row)

Time: 2ms total (execution 1ms / network 0ms)

demo@127.0.0.1:26257/demoapp/defaultdb> select * from system.statement_diagnostics;
id | statement_fingerprint | statement | collected_at | trace | bundle_chunks | error
---------------------+---------------------------------------------------------------+---------------------------------------+-------------------------------+-------+----------------------+--------
912968924944007169 | EXPLAIN ANALYZE (DEBUG) SELECT * FROM ij WHERE (j->'_') = '_' | SELECT * FROM ij WHERE (j->'x') = '1' | 2023-10-30 17:20:04.924363+00 | NULL | {912968924934602753} | NULL
(1 row)

Time: 1ms total (execution 1ms / network 0ms)

demo@127.0.0.1:26257/demoapp/defaultdb> \statement-diag download 912968924944007169
Bundle saved to "stmt-bundle-912968924944007169.zip"
```

We do get the statement, the environment, and the trace. But we do not get schemas or stats:

```
# michae2@michae2-crl-mbp ~crdb % unzip stmt-bundle-912968924944007169 -d stmt-bundle-912968924944007169
Archive: stmt-bundle-912968924944007169.zip
inflating: stmt-bundle-912968924944007169/statement.sql
inflating: stmt-bundle-912968924944007169/opt.txt
inflating: stmt-bundle-912968924944007169/opt-v.txt
inflating: stmt-bundle-912968924944007169/opt-vv.txt
inflating: stmt-bundle-912968924944007169/plan.txt
inflating: stmt-bundle-912968924944007169/distsql.html
inflating: stmt-bundle-912968924944007169/trace.json
inflating: stmt-bundle-912968924944007169/trace.txt
inflating: stmt-bundle-912968924944007169/trace-jaeger.json
inflating: stmt-bundle-912968924944007169/env.sql
inflating: stmt-bundle-912968924944007169/schema.sql
inflating: stmt-bundle-912968924944007169/errors.txt

# michae2@michae2-crl-mbp ~crdb % cd stmt-bundle-912968924944007169

# michae2@michae2-crl-mbp ~crdb/stmt-bundle-912968924944007169 % ls
distsql.html env.sql errors.txt opt-v.txt opt-vv.txt opt.txt plan.txt schema.sql statement.sql trace-jaeger.json trace.json trace.txt

# michae2@michae2-crl-mbp ~crdb/stmt-bundle-912968924944007169 % cat schema.sql
-- no schema collected
```

We should try to also get the schemas and stats.

Jira issue: CRDB-32861
Epic CRDB-37757

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.