Exceptions cannot be caught in the multi-threaded executor

Bug report

Required Info:

• Operating System:
  • Debian Bullseye (aarch64)
• Installation type:
  • From source
• Version or commit hash:
  • humble
• DDS implementation:
  • rmw_cyclonedds_cpp
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

We have a node at hand which provides services which are regularly called by other instances of this node via wireless interfaces. When these services are not responding fast enough pending requests will be removed. Sporadically we then see such nodes terminating with an rclcpp::exceptions::RCLError where the error message is set to failed to send response: error not set. We cannot catch that exception as far as we can tell.

Expected behavior

take_and_do_error_handling catches RCLError when handling the service.

Actual behavior

The corresponding executor thread terminates with the following backtrace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000fffff6485aa0 in __GI_abort () at abort.c:79
#2  0x0000fffff669e238 in __gnu_cxx::__verbose_terminate_handler() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x0000fffff669bd4c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000fffff669bdb0 in std::terminate() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x0000fffff669bd38 in std::rethrow_exception(std::__exception_ptr::exception_ptr) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000fffff71a6058 in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rcutils_error_state_s const*, void (*)()) () from /opt/ros/humble/lib/librclcpp.so
#7  0x0000fffff7e67d44 in rclcpp::Service<foobar_msgs::srv::AgentMapSrv>::send_response(rmw_request_id_s&, foobar_msgs::srv::AgentMapSrv_Response_<std::allocator<void> >&) () from /home/foobar/ros2_devel/lib/libfoobar_component.so
#8  0x0000fffff7e600e4 in rclcpp::Service<foobar_msgs::srv::AgentMapSrv>::handle_request(std::shared_ptr<rmw_request_id_s>, std::shared_ptr<void>) () from /home/foobar/ros2_devel/lib/libfoobar_component.so
#9  0x0000fffff71af514 in rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}::operator()() const
    () from /opt/ros/humble/lib/librclcpp.so
#10 0x0000fffff71b47b8 in void std::__invoke_impl<void, rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}&>(std::__invoke_other, rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}&) ()
   from /opt/ros/humble/lib/librclcpp.so
#11 0x0000fffff71b37a0 in std::enable_if<is_invocable_r_v<void, rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}&>, void>::type std::__invoke_r<void, rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}&>(rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}&) ()
   from /opt/ros/humble/lib/librclcpp.so
#12 0x0000fffff71b2130 in std::_Function_handler<void (), rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>)::{lambda()#2}>::_M_invoke(std::_Any_data const&) () from /opt/ros/humble/lib/librclcpp.so
#13 0x0000fffff7190638 in std::function<void ()>::operator()() const () from /opt/ros/humble/lib/librclcpp.so
#14 0x0000fffff71ae63c in take_and_do_error_handling(char const*, char const*, std::function<bool ()>, std::function<void ()>) ()
   from /opt/ros/humble/lib/librclcpp.so
#15 0x0000fffff71af648 in rclcpp::Executor::execute_service(std::shared_ptr<rclcpp::ServiceBase>) ()
   from /opt/ros/humble/lib/librclcpp.so
#16 0x0000fffff71ae3b0 in rclcpp::Executor::execute_any_executable(rclcpp::AnyExecutable&) () from /opt/ros/humble/lib/librclcpp.so
#17 0x0000fffff71bc554 in rclcpp::executors::MultiThreadedExecutor::run(unsigned long) () from /opt/ros/humble/lib/librclcpp.so
#18 0x0000fffff71be314 in void std::__invoke_impl<void, void (rclcpp::executors::MultiThreadedExecutor::*&)(unsigned long), rclcpp::executors::MultiThreadedExecutor*&, unsigned long&>(std::__invoke_memfun_deref, void (rclcpp::executors::MultiThreadedExecutor::*&)(unsigned long), rclcpp::executors::MultiThreadedExecutor*&, unsigned long&) () from /opt/ros/humble/lib/librclcpp.so
#19 0x0000fffff71be20c in std::__invoke_result<void (rclcpp::executors::MultiThreadedExecutor::*&)(unsigned long), rclcpp::executors::MultiThreadedExecutor*&, unsigned long&>::type std::__invoke<void (rclcpp::executors::MultiThreadedExecutor::*&)(unsigned long), rclcpp::executors::MultiThreadedExecutor*&, unsigned long&>(void (rclcpp::executors::MultiThreadedExecutor::*&)(unsigned long), rclcpp::executors::MultiThreadedExecutor*&, unsigned long&) () from /opt/ros/humble/lib/librclcpp.so
#20 0x0000fffff71be118 in void std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) ()
   from /opt/ros/humble/lib/librclcpp.so
#21 0x0000fffff71be07c in void std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)>::operator()<, void>() () from /opt/ros/humble/lib/librclcpp.so
#22 0x0000fffff71be02c in void std::__invoke_impl<void, std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executor--Type <RET> for more, q to quit, c to continue without paging--
s::MultiThreadedExecutor*, unsigned long))(unsigned long)>>(std::__invoke_other, std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)>&&) () from /opt/ros/humble/lib/librclcpp.so
#23 0x0000fffff71bdfbc in std::__invoke_result<std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)>>::type std::__invoke<std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)>>(std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)>&&) () from /opt/ros/humble/lib/librclcpp.so
#24 0x0000fffff71bdf58 in void std::thread::_Invoker<std::tuple<std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
   from /opt/ros/humble/lib/librclcpp.so
#25 0x0000fffff71bdf2c in std::thread::_Invoker<std::tuple<std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)> > >::operator()() () from /opt/ros/humble/lib/librclcpp.so
#26 0x0000fffff71bdf0c in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (rclcpp::executors::MultiThreadedExecutor::*(rclcpp::executors::MultiThreadedExecutor*, unsigned long))(unsigned long)> > > >::_M_run() ()
   from /opt/ros/humble/lib/librclcpp.so
#27 0x0000fffff66c6cac in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#28 0x0000fffff5bec648 in start_thread (arg=0xffffd174d440) at pthread_create.c:477
#29 0x0000fffff6536fdc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Start by tracing rclcpp::Executor::execute_service through take_and_do_error_handling, then inspect Service::send_response and MultiThreadedExecutor::run using the reported backtrace. Reproduce the service-response failure and verify that the RCLError is handled without terminating the executor thread.