[BUG]: Many pybind11 tests fail under ASan/UBSan

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

In user code I have been observing lots of segfaults and UB; naturally I tried to discover its cause via ASan and UBSan. After discovering issue #3514 it was unclear to me where the fixes should go: In user code or in pybind11. Hence I ran the pybind11 unit tests after a compilation with:

cmake -DDOWNLOAD_CATCH=ON  -DCMAKE_CXX_FLAGS="-fsanitize=address -fsanitize=undefined -g" ../ -G Ninja

The following tests failed:

~/pybind11/build/tests ❯❯❯ python3 -m pytest ../../tests/ -v
../../tests/test_builtin_casters.py::test_single_char_arguments 
~/pybind11/include/pybind11/detail/internals.h:391:46: runtime error: member call on address 0x000130eb9540 which does not point to an object of type 'pybind11::builtin_exception'
0x000130eb9540: note: object is of type 'pybind11::value_error'
 01 00 00 00  88 75 9e 17 01 00 00 00  28 78 16 3a 01 00 00 00  dd dd dd dd dd dd dd dd  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'pybind11::value_error'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior 
~/pybind11/include/pybind11/detail/internals.h:391:46 in
PASSED                                                                                 [  2%]
../../tests/test_local_bindings.py::test_stl_bind_local 
~/pybind11/include/pybind11/detail/internals.h:391:46: runtime error: member call on address 0x0001134b83c0 which does not point to an object of type 'pybind11::builtin_exception'
0x0001134b83c0: note: object is of type 'pybind11::stop_iteration'
 01 00 00 00  c8 ca 9f 17 01 00 00 00  18 e4 37 0a 01 00 00 00  dd dd dd dd dd dd dd dd  00 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'pybind11::stop_iteration'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior 
~/pybind11/include/pybind11/detail/internals.h:391:46 in
PASSED                                                                                         [ 51%]
../../tests/test_numpy_dtypes.py::test_recarray ~/pybind11/tests/test_numpy_dtypes.cpp:56:30: runtime error: reference binding to misaligned address 0x000147c4fc29 for type 'const SimpleStruct', which requires 8 byte alignment
0x000147c4fc29: note: pointer points here
 00 00 04  c0 01 be be be 01 00 00  00 00 00 c0 3f be be be  be 00 00 00 00 00 00 04  c0 00 02 00 00
              ^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior 
~/pybind11/tests/test_numpy_dtypes.cpp:56:30 in
FAILED                                                                                                 [ 71%]

Note that the python interpreter must be source compiled with ./configure --with-address-sanitizer --with-pydebug.

Reproducible example code

In the repo.