[Bug] smallsort panic in tokio task

Bug report

Getting these logs on latest version of node, node is crashlooping. I can't pinpoint particular subgraph because it never has same logs preceeding the panic.

Relevant log output

thread 'tokio-runtime-worker' panicked at library/core/src/slice/sort/shared/smallsort.rs:860:5:
user-provided comparison function does not correctly implement a total order
stack backtrace:
   0:     0x564e9f20f032 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc04c8f544ab24d66
   1:     0x564e9f239b03 - core::fmt::write::hfe57b7174b7d8eab
   2:     0x564e9f20ae13 - std::io::Write::write_fmt::h154385efa8565236
   3:     0x564e9f20ee82 - std::sys::backtrace::BacktraceLock::print::h0c8f24e22f5873a8
   4:     0x564e9f20fee2 - std::panicking::default_hook::{{closure}}::hd07d57e6a602c8e4
   5:     0x564e9f20fce5 - std::panicking::default_hook::h63d12f7d95bd91ed
   6:     0x564e9f210882 - std::panicking::rust_panic_with_hook::h33b18b24045abff4
   7:     0x564e9f2105f6 - std::panicking::begin_panic_handler::{{closure}}::hf8313cc2fd0126bc
   8:     0x564e9f20f539 - std::sys::backtrace::__rust_end_short_backtrace::h57fe07c8aea5c98a
   9:     0x564e9f2102bd - __rustc[95feac21a9532783]::rust_begin_unwind
  10:     0x564e9daa12e0 - core::panicking::panic_fmt::hd54fb667be51beea
  11:     0x564e9daa174b - core::slice::sort::shared::smallsort::panic_on_ord_violation::h4d7d0ae5a308eb96
  12:     0x564e9e5f2789 - core::slice::sort::shared::smallsort::bidirectional_merge::h1044bbad7b7beb44
  13:     0x564e9e5f3dc7 - core::slice::sort::shared::smallsort::small_sort_general_with_scratch::h43437cc608b24fb8
  14:     0x564e9e5d67ca - core::slice::sort::stable::quicksort::quicksort::h6680c382a9895184
  15:     0x564e9e5c292a - core::slice::sort::stable::drift::sort::hd5b4358f62f7bc88
  16:     0x564e9e6817a3 - core::slice::sort::stable::driftsort_main::hb989dc57d61dcb54
  17:     0x564e9e5c31f3 - graph::blockchain::block_stream::BlockWithTriggers<C>::new_with_triggers::h382c15402379671a
  18:     0x564e9e5c3377 - graph::blockchain::block_stream::BlockWithTriggers<C>::new::h9f977ef62ae27e9e
  19:     0x564e9e627127 - <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold::h16cd43443c2189d5
  20:     0x564e9e6f22e0 - <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold::h8517dd5b89e876ea
  21:     0x564e9e68388f - alloc::vec::Vec<T,A>::extend_desugared::h022b117deec25a99
  22:     0x564e9e675dbb - <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter::h082fdbcf4f5694b8
  23:     0x564e9e6f7aae - core::iter::adapters::try_process::h524dee65de37f739
  24:     0x564e9e6e073c - graph_chain_ethereum::ethereum_adapter::blocks_with_triggers::{{closure}}::hbcddd29392b358df
  25:     0x564e9e6e8721 - <graph_chain_ethereum::chain::TriggersAdapter as graph::blockchain::block_stream::TriggersAdapter<graph_chain_ethereum::chain::Chain>>::scan_triggers::{{closure}}::hd19e36313c94865c
  26:     0x564e9e5fcf4e - <graph::blockchain::polling_block_stream::PollingBlockStream<C> as futures_core::stream::Stream>::poll_next::{{closure}}::hd37678890ff68cbd
  27:     0x564e9e5fb564 - <graph::blockchain::polling_block_stream::PollingBlockStream<C> as futures_core::stream::Stream>::poll_next::h4b4aca241f62d2f1
  28:     0x564e9de02fea - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll::h8b4edb2df8bc6275
  29:     0x564e9db12930 - std::panic::catch_unwind::h366e6067c275c748
  30:     0x564e9de602ba - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hb6404ec87feab6b4
  31:     0x564e9db9415b - tokio::runtime::task::core::Core<T,S>::poll::hdbda6337e5b585e6
  32:     0x564e9db13f24 - std::panic::catch_unwind::h686a3f3ae1ca0632
  33:     0x564e9dec2f15 - tokio::runtime::task::harness::Harness<T,S>::poll::h624f0d6501408bad
  34:     0x564e9f09c9e6 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hd4150d71a835da8f
  35:     0x564e9f09c260 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h4ca0d2f167d185e8
  36:     0x564e9f0942ad - tokio::runtime::context::scoped::Scoped<T>::set::h6bb32a3dcf532dbc
  37:     0x564e9f096ae1 - tokio::runtime::context::runtime::enter_runtime::h73863383e535eab2
  38:     0x564e9f09bd6b - tokio::runtime::scheduler::multi_thread::worker::run::he18d531dba0987c7
  39:     0x564e9f099835 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h8649ce37c791bb9d
  40:     0x564e9f08c0a1 - tokio::runtime::task::core::Core<T,S>::poll::hdb1e4176d9a11895
  41:     0x564e9f086e51 - std::panic::catch_unwind::h2168414212ee0f61
  42:     0x564e9f086b5e - tokio::runtime::task::harness::poll_future::h4dd9e57c4da4e573
  43:     0x564e9f085f48 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hf0009ae66083b8b7
  44:     0x564e9f085cc4 - tokio::runtime::task::harness::Harness<T,S>::poll::h1ea3c48f1483ded5
  45:     0x564e9f089b2c - tokio::runtime::task::UnownedTask<S>::run::h581e0dde7d69b0a4
  46:     0x564e9f08afeb - tokio::runtime::blocking::pool::Inner::run::h5016cef15e1d031a
  47:     0x564e9f0874ae - std::sys::backtrace::__rust_begin_short_backtrace::h17d92e3a31e8f073
  48:     0x564e9f08f6db - core::ops::function::FnOnce::call_once{{vtable.shim}}::hc9533e2ba1f167ac
  49:     0x564e9f21291b - std::sys::pal::unix::thread::Thread::new::thread_start::h2717d4f9aca577e6
  50:     0x7f3ef43051c4 - <unknown>
  51:     0x7f3ef4384ac0 - __clone
  52:                0x0 - <unknown>
Panic in tokio task, aborting!

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux

Bắt đầu tại graph::blockchain::block_stream::BlockWithTriggers::new_with_triggers và graph_chain_ethereum::ethereum_adapter::blocks_with_triggers, nơi stack cho thấy panic xảy ra, rồi kiểm tra đường đi sắp xếp trigger. Tạo một subgraph hoặc đầu vào có thể tái hiện từ báo cáo crash, sau đó xác minh rằng việc xử lý nó không còn khiến node rơi vào crashloop.