quickwit-oss / quickwit-oss/quickwit

DeleteTaskPlanner can stacktrace if an index is deleted before it passes

Open
#5,045 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
11.7k
Forks
597
Avg merge
2d 22h
Merged PRs (30d)
37

Description

i ran the integration test on a clean quickwit instance. Tests passed, but quickwit logged this backtrace:

logs 
2024-05-29T10:43:34.663Z ERROR quickwit_actors::spawn_builder: actor-failure cause=index `empty_index` not found

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
   2: quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner::send_delete_operations::{{closure}}
   3: <quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner as quickwit_actors::actor::Handler<quickwit_janitor::actors::delete_task_planner::PlanDeleteOperations>>::handle::{{closure}}
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
   5: <quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner as quickwit_actors::actor::Handler<quickwit_janitor::actors::delete_task_planner::PlanDeleteLoop>>::handle::{{closure}}
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
   7: <H as quickwit_actors::actor::DeferableReplyHandler<M>>::handle_message::{{closure}}
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
   9: <core::option::Option<(tokio::sync::oneshot::Sender<<A as quickwit_actors::actor::DeferableReplyHandler<M>>::Reply>,M)> as quickwit_actors::envelope::EnvelopeT<A>>::handle_message::{{closure}}
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
  11: quickwit_actors::envelope::Envelope<A>::handle_message::{{closure}}
  12: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_one_message::{{closure}}
  13: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_all_available_messages::{{closure}}
  14: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_messages::{{closure}}
  15: quickwit_actors::spawn_builder::actor_loop::{{closure}}
  16: quickwit_actors::spawn_builder::SpawnBuilder<A>::spawn::{{closure}}
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
  18: tokio::runtime::task::core::Core<T,S>::poll
  19: tokio::runtime::task::harness::poll_future::{{closure}}
  20: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  21: std::panicking::try::do_call
  22: __rust_try
  23: std::panicking::try
  24: std::panic::catch_unwind
  25: tokio::runtime::task::harness::poll_future
  26: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  27: tokio::runtime::task::harness::Harness<T,S>::poll
  28: tokio::runtime::task::raw::poll
  29: tokio::runtime::task::raw::RawTask::poll
  30: tokio::runtime::task::LocalNotified<S>::run
  31: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  33: tokio::runtime::scheduler::multi_thread::worker::Context::run
  34: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
  35: tokio::runtime::context::scoped::Scoped<T>::set
  36: tokio::runtime::context::set_scheduler::{{closure}}
  37: std::thread::local::LocalKey<T>::try_with
  38: std::thread::local::LocalKey<T>::with
  39: tokio::runtime::context::set_scheduler
  40: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
  41: tokio::runtime::context::runtime::enter_runtime
  42: tokio::runtime::scheduler::multi_thread::worker::run
  43: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
  44: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  45: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
  46: tokio::runtime::task::core::Core<T,S>::poll
  47: tokio::runtime::task::harness::poll_future::{{closure}}
  48: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  49: std::panicking::try::do_call
  50: __rust_try
  51: std::panicking::try
  52: std::panic::catch_unwind
  53: tokio::runtime::task::harness::poll_future
  54: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  55: tokio::runtime::task::harness::Harness<T,S>::poll
  56: tokio::runtime::task::raw::poll
  57: tokio::runtime::task::raw::RawTask::poll
  58: tokio::runtime::task::UnownedTask<S>::run
  59: tokio::runtime::blocking::pool::Task::run
  60: tokio::runtime::blocking::pool::Inner::run
  61: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
  62: std::sys_common::backtrace::__rust_begin_short_backtrace
  63: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
  64: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  65: std::panicking::try::do_call
  66: __rust_try
  67: std::panicking::try
  68: std::thread::Builder::spawn_unchecked_::{{closure}}
  69: core::ops::function::FnOnce::call_once{{vtable.shim}}
  70: std::sys::pal::unix::thread::Thread::new::thread_start
  71: <unknown>
  72: <unknown> exit_status=Failure(index `empty_index` not found

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
   2: quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner::send_delete_operations::{{closure}}
   3: <quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner as quickwit_actors::actor::Handler<quickwit_janitor::actors::delete_task_planner::PlanDeleteOperations>>::handle::{{closure}}
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
   5: <quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner as quickwit_actors::actor::Handler<quickwit_janitor::actors::delete_task_planner::PlanDeleteLoop>>::handle::{{closure}}
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
   7: <H as quickwit_actors::actor::DeferableReplyHandler<M>>::handle_message::{{closure}}
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
   9: <core::option::Option<(tokio::sync::oneshot::Sender<<A as quickwit_actors::actor::DeferableReplyHandler<M>>::Reply>,M)> as quickwit_actors::envelope::EnvelopeT<A>>::handle_message::{{closure}}
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
  11: quickwit_actors::envelope::Envelope<A>::handle_message::{{closure}}
  12: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_one_message::{{closure}}
  13: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_all_available_messages::{{closure}}
  14: quickwit_actors::spawn_builder::ActorExecutionEnv<A>::process_messages::{{closure}}
  15: quickwit_actors::spawn_builder::actor_loop::{{closure}}
  16: quickwit_actors::spawn_builder::SpawnBuilder<A>::spawn::{{closure}}
  17: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
  18: tokio::runtime::task::core::Core<T,S>::poll
  19: tokio::runtime::task::harness::poll_future::{{closure}}
  20: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  21: std::panicking::try::do_call
  22: __rust_try
  23: std::panicking::try
  24: std::panic::catch_unwind
  25: tokio::runtime::task::harness::poll_future
  26: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  27: tokio::runtime::task::harness::Harness<T,S>::poll
  28: tokio::runtime::task::raw::poll
  29: tokio::runtime::task::raw::RawTask::poll
  30: tokio::runtime::task::LocalNotified<S>::run
  31: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
  32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  33: tokio::runtime::scheduler::multi_thread::worker::Context::run
  34: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
  35: tokio::runtime::context::scoped::Scoped<T>::set
  36: tokio::runtime::context::set_scheduler::{{closure}}
  37: std::thread::local::LocalKey<T>::try_with
  38: std::thread::local::LocalKey<T>::with
  39: tokio::runtime::context::set_scheduler
  40: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
  41: tokio::runtime::context::runtime::enter_runtime
  42: tokio::runtime::scheduler::multi_thread::worker::run
  43: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
  44: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  45: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
  46: tokio::runtime::task::core::Core<T,S>::poll
  47: tokio::runtime::task::harness::poll_future::{{closure}}
  48: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  49: std::panicking::try::do_call
  50: __rust_try
  51: std::panicking::try
  52: std::panic::catch_unwind
  53: tokio::runtime::task::harness::poll_future
  54: tokio::runtime::task::harness::Harness<T,S>::poll_inner
  55: tokio::runtime::task::harness::Harness<T,S>::poll
  56: tokio::runtime::task::raw::poll
  57: tokio::runtime::task::raw::RawTask::poll
  58: tokio::runtime::task::UnownedTask<S>::run
  59: tokio::runtime::blocking::pool::Task::run
  60: tokio::runtime::blocking::pool::Inner::run
  61: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
  62: std::sys_common::backtrace::__rust_begin_short_backtrace
  63: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
  64: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
  65: std::panicking::try::do_call
  66: __rust_try
  67: std::panicking::try
  68: std::thread::Builder::spawn_unchecked_::{{closure}}
  69: core::ops::function::FnOnce::call_once{{vtable.shim}}
  70: std::sys::pal::unix::thread::Thread::new::thread_start
  71: <unknown>
  72: <unknown>)

this looks like a race condition where the delete task planner tries to process an index which just got delete. It's probably harmless, but very verbose.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The stack trace points to quickwit_janitor::actors::delete_task_planner::DeleteTaskPlanner::send_delete_operations. Start by reading that entry point and reproduce the integration test on a clean Quickwit instance while deleting an index before it is processed. Done means the race no longer produces a verbose actor failure or stacktrace.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.