awslabs / awslabs/shuttle

`clock.rs:19:29: attempt to subtract with overflow`

Open
#192 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.1k
Forks
59
Avg merge
4d 2h
Merged PRs (30d)
15

Description

Thanks for your work on shuttle. It helped us to identify (and fix) many multithreading issues in salsa.

I'm now trying to write a new test but I run into an immediate panic. I'm sorry for not having a better repro.

```rust
let mut db = DatabaseImpl::default();

let input = Input::new(&db, 1);
let result = query_b(&db, input);

// assert_eq!(result, MAX);

input.set_value(&mut db).to(2);

let db_t1 = db.clone();
// let db_t2 = db.clone();
let db_t3 = db;

let t1 = thread::spawn(move || query_a(&db_t1, input));
// let t2 = thread::spawn(move || query_b(&db_t2, input));
// let t3 = thread::spawn(move || query_c(&db_t3, input));

let r_t1 = t1.join().unwrap();
// let r_t2 = t2.join().unwrap();
// let r_t3 = t3.join().unwrap();

// assert_eq!((r_t1, r_t2, r_t3), (MAX, MAX, MAX));
```

This test passes with shuttle.

But shuttle panics as soon as I start `t2` or `t3`,

```rust
let mut db = DatabaseImpl::default();

let input = Input::new(&db, 1);
let result = query_b(&db, input);

// assert_eq!(result, MAX);

input.set_value(&mut db).to(2);

let db_t1 = db.clone();
// let db_t2 = db.clone();
let db_t3 = db;

let t1 = thread::spawn(move || query_a(&db_t1, input));
let t2 = thread::spawn(move || query_b(&db_t2, input));
// let t3 = thread::spawn(move || query_c(&db_t3, input));

let r_t1 = t1.join().unwrap();
let r_t2 = t2.join().unwrap();
// let r_t3 = t3.join().unwrap();

// assert_eq!((r_t1, r_t2, r_t3), (MAX, MAX, MAX));
```

But the test also passes if i comment out `query_b`. Now, `query_b` doesn't spawn any threads but we use thread local state, the current thread id and various locks.

```
thread 'cycle_nested_three_threads_changed::the_test' panicked at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/task/clock.rs:19:29:
attempt to subtract with overflow
stack backtrace:
0: rust_begin_unwind
at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:695:5
1: core::panicking::panic_fmt
at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panicking.rs:75:14
2: core::panicking::panic_const::panic_const_sub_overflow
at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panicking.rs:178:21
3: shuttle::runtime::task::clock::VectorClock::extend
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/task/clock.rs:19:29
4: shuttle::runtime::execution::ExecutionState::spawn_thread::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/execution.rs:428:13
5: shuttle::runtime::execution::ExecutionState::try_with::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/execution.rs:329:26
6: scoped_tls::ScopedKey::with
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/scoped-tls-1.0.1/src/lib.rs:171:13
7: shuttle::runtime::execution::ExecutionState::try_with
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/execution.rs:327:13
8: shuttle::runtime::execution::ExecutionState::with
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/execution.rs:316:9
9: shuttle::runtime::execution::ExecutionState::spawn_thread
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/execution.rs:415:23
10: shuttle::thread::spawn_named_unchecked
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/thread.rs:178:9
11: shuttle::thread::spawn_named
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/thread.rs:158:14
12: shuttle::thread::spawn
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/thread.rs:147:5
13: parallel::cycle_nested_three_threads_changed::the_test::{{closure}}
at ./tests/parallel/cycle_nested_three_threads_changed.rs:84:18
14: shuttle::runtime::runner::Runner::run::{{closure}}::{{closure}}::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/runner.rs:106:101
15: shuttle::thread::thread_fn
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/thread.rs:201:15
16: shuttle::runtime::execution::Execution::run::{{closure}}::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/execution.rs:83:34
17: core::ops::function::FnOnce::call_once{{vtable.shim}}
at /Users/micha/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
18: as core::ops::function::FnOnce>::call_once
at /Users/micha/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1976:9
19: shuttle::runtime::thread::continuation::Continuation::new::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/shuttle-0.8.1/src/runtime/thread/continuation.rs:87:21
20: generator::gen_impl::GeneratorImpl::init_code::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/gen_impl.rs:357:21
21: generator::stack::StackBox::call_once
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/stack/mod.rs:137:13
22: generator::stack::Func::call_once
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/stack/mod.rs:119:9
23: generator::detail::gen::gen_init_impl::{{closure}}
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/detail/gen.rs:35:9
24: core::ops::function::FnOnce::call_once
at /Users/micha/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
25: std::panicking::try::do_call
at /Users/micha/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:587:40
26: ___rust_try
27: std::panicking::try
at /Users/micha/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panicking.rs:550:19
28: std::panic::catch_unwind
at /Users/micha/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/std/src/panic.rs:358:14
29: generator::detail::gen::catch_unwind_filter
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/detail/gen.rs:23:5
30: generator::detail::gen::gen_init_impl
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/detail/gen.rs:50:25
31: generator::detail::asm::gen_init
at /Users/micha/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/generator-0.8.5/src/detail/aarch64_unix.rs:18:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```

Contributor guide

Open the contributing guide

Research direction

Start with runtime/task/clock.rs, especially VectorClock::extend, and trace its call from runtime/execution.rs in spawn_thread. Reproduce the panic with tests/parallel/cycle_nested_three_threads_changed.rs at the thread-spawn location shown in the trace. Done means the reported multi-thread test no longer panics with subtraction overflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.