Saving backtrace
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
I think it would be really useful to log the backtrace in some cases:
- When we get an error
- When we get an assertion failure
- when we get a crash
1-> When we get an error, we have no way of knowing where the error comes from, hence we will need to attach a debugger and try to reproduce the error to find what is the codepath that is causing the error sometimes. However it might be hard to reproduce the error. If the error happens in a user cluster, if they had the backtrace along with the error, I believe it would save us a lot of time. This could be controlled via a GUC so that we can set what level of logging will include backtracing as well. By default it could be 'ERROR'.
2-> Postgres 13 has added backtrace support for assertion failures:
https://github.com/postgres/postgres/blob/master/src/backend/utils/error/assert.c#L48-L56
with the commit: https://github.com/postgres/postgres/commit/71a8a4f6e36547bb060dbcc961ea9b57420f7190
So by defining the `HAVE_BACKTRACE_SYMBOLS` macro we will already have backtrace for assertion failures.
3-> When customers get a crash, we usually ask for their coredump, I guess even though it doesn't replace coredump, it would still be beneficial to log backtrace prior to exiting with a signal handler. For example we can register a SIGSEV handler and just log the backtrace there. The handler shouldn't have anything that could crash I guess. For example `backtrace_symbols` has malloc, so it might not be safe to use here.
As an initial step, we can use the `backtrace` function to get the backtrace and `backtrace_symbols` to get backtrace lines then we can append this to error data for number 1, or we can use `backtrace_symbols_fd` to log it. We might also need to add the `-rdynamic` to the linker options so that we have more function names available in the backtrace, however the static functions won't have their name in the backtrace ( just their address), and we won't have the line numbers.
Some basic research suggests that https://github.com/ianlancetaylor/libbacktrace is a good library for having better backtraces. I guess one option could be to add this to our vendor folder, this seems to have line numbers as well as function names for static functions.
An example backtrace with `backtrace_symbols`:

The same error with libbacktrace:

Contributor guide
Assessment
This issue has not been assessed yet.