citusdata / citusdata/citus

Saving backtrace

Open
#3,979 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

I think it would be really useful to log the backtrace in some cases:
- When we get an error
- When we get an assertion failure
- when we get a crash

1-> When we get an error, we have no way of knowing where the error comes from, hence we will need to attach a debugger and try to reproduce the error to find what is the codepath that is causing the error sometimes. However it might be hard to reproduce the error. If the error happens in a user cluster, if they had the backtrace along with the error, I believe it would save us a lot of time. This could be controlled via a GUC so that we can set what level of logging will include backtracing as well. By default it could be 'ERROR'.

2-> Postgres 13 has added backtrace support for assertion failures:
https://github.com/postgres/postgres/blob/master/src/backend/utils/error/assert.c#L48-L56
with the commit: https://github.com/postgres/postgres/commit/71a8a4f6e36547bb060dbcc961ea9b57420f7190
So by defining the `HAVE_BACKTRACE_SYMBOLS` macro we will already have backtrace for assertion failures.

3-> When customers get a crash, we usually ask for their coredump, I guess even though it doesn't replace coredump, it would still be beneficial to log backtrace prior to exiting with a signal handler. For example we can register a SIGSEV handler and just log the backtrace there. The handler shouldn't have anything that could crash I guess. For example `backtrace_symbols` has malloc, so it might not be safe to use here.

As an initial step, we can use the `backtrace` function to get the backtrace and `backtrace_symbols` to get backtrace lines then we can append this to error data for number 1, or we can use `backtrace_symbols_fd` to log it. We might also need to add the `-rdynamic` to the linker options so that we have more function names available in the backtrace, however the static functions won't have their name in the backtrace ( just their address), and we won't have the line numbers.

Some basic research suggests that https://github.com/ianlancetaylor/libbacktrace is a good library for having better backtraces. I guess one option could be to add this to our vendor folder, this seems to have line numbers as well as function names for static functions.

An example backtrace with `backtrace_symbols`:
![image](https://user-images.githubusercontent.com/17179875/88024695-ccefd480-cb3b-11ea-8f6f-b3b3aaa6d22c.png)

The same error with libbacktrace:

![image](https://user-images.githubusercontent.com/17179875/88024771-ea24a300-cb3b-11ea-94c4-709dfefd701f.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.