Proposal: Automatically print backtrace of crashing backends in test outputs
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Often we make large changes we hit some crashes when running `check-multi`. To get the backtrace of the where program crashes, we can try manually running the test in a backend that has gdb attached to, as described in https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD.
Given that in large projects we need to do this many times, this manual effort would add hours or days to the overall debugging process. Such large projects that I've encountered in last few months are: adapative executor, pg12 support, ...
It would be nicer if the regression tester runner could automatically attach gdb to the backends and automatically print backtrace on crash.
A simple prototype implementation of this is https://github.com/citusdata/citus/pull/3093, which uses `pg_regress` `--launcher` option to use a gdb backed psql instead of the normal psql. This gdb backed psql, opens a psql process, sends "SELECT pg_backend_pid()" to it, and opens a gdb connection to it. Then continues normally by redirecting stdin/stdout/stderr to/from the psql process. On crash (exit code != 0), it prints the output of the gdb process which includes the backtrace.
The above approach has limitations:
- It won't print backtraces for after `\connect` as the backend pid changes.
- It won't print backtraces for workers or maintenance daemon.
Other alternatives might be much more complicated, and this approach might be good enough for most of our needs.
Contributor guide
Research direction
Start by reviewing the check-multi workflow, pg_regress's --launcher option, and the prototype in citusdata/citus#3093. Run the relevant regression tests and determine whether crashing backends produce backtraces, including the stated limitations around \connect, workers, and maintenance daemons.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, postgresql
- Domain
- databases, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100