Message Filter stuck dropping stale messages due to large queue size

Bug report

Required Info:

• Operating System:
  Ubuntu 18.04
• Installation type:
  Source
• Version or commit hash:
• DDS implementation:
  cyclone
• Client library (if applicable):
  rclcpp

Very rarely, my system will enter an error state where all of my message filters start rejecting my messages.
My log output ends up looking like this until I manually kill my nodes:

[1610551896.439708854] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551894.931 for reason 'Unknown'
[1610551896.472888675] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551894.960 for reason 'Unknown'
[1610551896.505286865] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551894.990 for reason 'Unknown'
[1610551896.536371604] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.020 for reason 'Unknown'
[1610551896.560983603] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.050 for reason 'Unknown'
[1610551896.587268162] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.080 for reason 'Unknown'
[1610551896.619227590] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.110 for reason 'Unknown'
[1610551896.654364589] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.140 for reason 'Unknown'
[1610551896.677907253] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.170 for reason 'Unknown'
[1610551896.704027512] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.201 for reason 'Unknown'
[1610551896.742694255] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.230 for reason 'Unknown'
[1610551896.762921561] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.260 for reason 'Unknown'
[1610551896.794901727] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.290 for reason 'Unknown'
[1610551896.820790032] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.320 for reason 'Unknown'
[1610551896.855324540] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.350 for reason 'Unknown'
[1610551896.889219195] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.383 for reason 'Unknown'
[1610551896.914657912] [local_costmap.local_costmap_rclcpp_node]: Message Filter dropping message: frame 'camera_link' at time 1610551895.410 for reason 'Unknown'

I've confirmed that my transforms are still being published.

Taking a look at the implementation, my problem seems to be at the transform check. Something causes my message filter queue to fill up. My queue depth is so large that the last message in the queue is too old to find a valid transform. Because only one message in the queue is processed per transform, and the sensor topic publishes at the same rate as my transforms, the oldest message in my queue continues to be stale.

I'm wondering why the Message Filter only processes one message per transform? My thinking is that if as many messages as possible were processed on every transform arrival, then I could avoid this stale data at the end of my queue permanently disabling operation.