influxdata / influxdata/influxdb

Investigate intermittent test failure in write_buffer test

Open
#25,448 0 comments 0 reactions 0 assignees View on GitHub
v3
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

When running tests locally ran into the following test failure,

```
--- STDERR: influxdb3_write write_buffer::tests::last_cache_create_and_delete_is_durable ---
thread 'write_buffer::tests::last_cache_create_and_delete_is_durable' panicked at influxdb3_write/src/write_buffer/mod.rs:850:10:
called `Result::unwrap()` on an `Err` value: WalError(ObjectStoreError(NotFound { path: "test_host/wal/00000000004.wal", source: NoDataInMemory { path: "test_host/wal/00000000004.wal" } }))
stack backtrace:
0: rust_begin_unwind
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/panicking.rs:652:5
1: core::panicking::panic_fmt
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/panicking.rs:72:14
2: core::result::unwrap_failed
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/result.rs:1679:5
3: core::result::Result::unwrap
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/result.rs:1102:23
4: influxdb3_write::write_buffer::tests::last_cache_create_and_delete_is_durable::{{closure}}
at ./src/write_buffer/mod.rs:835:20
5: as core::future::future::Future>::poll
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/future/future.rs:123:9
6: as core::future::future::Future>::poll
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
7: as core::future::future::Future>::poll
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/future/future.rs:123:9
8: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:696:57
9: tokio::runtime::coop::with_budget
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
10: tokio::runtime::coop::budget
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
11: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:696:25
12: tokio::runtime::scheduler::current_thread::Context::enter
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:423:19
13: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:695:36
14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:774:68
15: tokio::runtime::context::scoped::Scoped::set
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
16: tokio::runtime::context::set_scheduler::{{closure}}
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
17: std::thread::local::LocalKey::try_with
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/thread/local.rs:283:12
18: std::thread::local::LocalKey::with
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/std/src/thread/local.rs:260:9
19: tokio::runtime::context::set_scheduler
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:9
20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:774:27
21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:683:19
22: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:191:28
23: tokio::runtime::context::runtime::enter_runtime
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
24: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:179:9
25: tokio::runtime::runtime::Runtime::block_on_inner
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:361:47
26: tokio::runtime::runtime::Runtime::block_on
at /home/praveen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:335:13
27: influxdb3_write::write_buffer::tests::last_cache_create_and_delete_is_durable
at ./src/write_buffer/mod.rs:852:9
28: influxdb3_write::write_buffer::tests::last_cache_create_and_delete_is_durable::{{closure}}
at ./src/write_buffer/mod.rs:704:55
29: core::ops::function::FnOnce::call_once
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:250:5
30: core::ops::function::FnOnce::call_once
at /rustc/051478957371ee0084a7c0913941d2a8c4757bb9/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

```

The same test has failed in CI as well,
https://app.circleci.com/pipelines/github/influxdata/influxdb/41703/workflows/da14ee4d-e712-42c2-a5b9-f5cba67c1af1/jobs/389885

Contributor guide

Open the contributing guide

Research direction

Start with influxdb3_write/src/write_buffer/mod.rs and the write_buffer::tests::last_cache_create_and_delete_is_durable test, which fails at the reported unwrap with a missing WAL object. Run the test locally and compare its behavior with the linked CircleCI failure; done means identifying and addressing the intermittent failure so the test passes reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.