cockroachdb / cockroachdb/cockroach
sql: some node crashes do not pass through sql.(*connExecutor).closeWrapper
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Node crashes caused by unrecovered panics during query execution pass through `sql.(*connExecutor).closeWrapper` which does a couple of nice things:
* decorates the error with the current statement
* logs the panic
* reports the panic to telemetry
We can use `crdb_internal.force_panic` to see an example:
```
demo@127.0.0.1:26257/demoapp/defaultdb> SELECT crdb_internal.force_panic('oops');
*
* ERROR: a SQL panic has occurred while executing the following statement:
* SELECT crdb_internal.force_panic('oops')
*
*
* ERROR: a panic has occurred!
* panic: oops
* (1) attached stack trace
* -- stack trace:
* | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
* | pkg/sql/conn_executor.go:1051
* | [...repeated from below...]
* Wraps: (2) while executing: SELECT crdb_internal.force_panic(_)
* Wraps: (3) attached stack trace
* -- stack trace:
* | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
* | pkg/sql/conn_executor.go:1051
* | runtime.gopanic
* | GOROOT/src/runtime/panic.go:791
* | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
* | pkg/sql/colexecerror/error.go:137
* | runtime.gopanic
* | GOROOT/src/runtime/panic.go:791
* | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
* | pkg/sql/colexecerror/error.go:137
* | runtime.gopanic
* | GOROOT/src/runtime/panic.go:791
* | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.NonCatchablePanic
* | pkg/sql/colexecerror/error.go:317
* | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func275
* | pkg/sql/sem/builtins/builtins.go:5935
...
```
Unfortunately, not all node crashes can be handled in this way. For example, stack overflows print "fatal error: stack overflow" to stderr and then immediately exit the process:
```
demo@127.0.0.1:26257/demoapp/defaultdb> SELECT crdb_internal.force_panic('oops', 'stackOverflow');
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x140294ec380 stack=[0x140294ec000, 0x140494ec000]
fatal error: stack overflow
runtime stack:
runtime.throw({0x10847946b?, 0x200000008?})
GOROOT/src/runtime/panic.go:1073 +0x38 fp=0x175792db0 sp=0x175792d80 pc=0x102fdc588
runtime.newstack()
GOROOT/src/runtime/stack.go:1117 +0x460 fp=0x175792ef0 sp=0x175792db0 pc=0x102fbe9f0
runtime.morestack()
src/runtime/asm_arm64.s:342 +0x70 fp=0x175792ef0 sp=0x175792ef0 pc=0x102fe3b10
goroutine 4279 gp=0x14005d36e00 m=18 mp=0x140015d0008 [running]:
github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func276.1(0xaaa986)
pkg/sql/sem/builtins/builtins.go:5977 +0x88 fp=0x140294ec380 sp=0x140294ec380 pc=0x1051b2c88
github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func276.1(0xaaa985)
pkg/sql/sem/builtins/builtins.go:5981 +0x48 fp=0x140294ec3b0 sp=0x140294ec380 pc=0x1051b2c48
github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func276.1(0xaaa984)
pkg/sql/sem/builtins/builtins.go:5981 +0x48 fp=0x140294ec3e0 sp=0x140294ec3b0 pc=0x1051b2c48
...
```
This lack of statement / logging / sentry report caused us some difficulty in #144020.
One idea for improving observability:
- add an in-memory log (maybe related to the flight recorder tracing?) with the current statements being executed and other current process info
- modify our own version of src/runtime/panic.go to dump this in-memory log on stack overflow and other fatal errors
This still wouldn't send the stacktrace through normal logging or telemetry, but at least we could find out the statement from stderr.
Jira issue: CRDB-50867
Contributor guide
Assessment
This issue has not been assessed yet.