apache / apache/arrow-rs-object-store
Intermittent failure with object_store aws::tests::s3_test emulator
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Describe the bug**
aws::tests::s3_test emulator CI tests are failing on main
**To Reproduce**
https://github.com/apache/arrow-rs/actions/runs/12672198886/job/35315696126
```
---- aws::tests::s3_test stdout ----
thread 'aws::tests::s3_test' panicked at src/integration.rs:960:5:
assertion failed: content_list.is_empty()
stack backtrace:
0: rust_begin_unwind
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5
3: object_store::integration::list_with_delimiter::{{closure}}
at ./src/integration.rs:960:5
4: object_store::aws::tests::s3_test::{{closure}}
at ./src/aws/mod.rs:569:43
5: as core::future::future::Future>::poll
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/future/future.rs:123:9
6: as core::future::future::Future>::poll
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/future/future.rs:123:9
7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:729:57
8: tokio::runtime::coop::with_budget
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/coop.rs:107:5
9: tokio::runtime::coop::budget
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/coop.rs:73:5
10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:729:25
11: tokio::runtime::scheduler::current_thread::Context::enter
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:428:19
12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:728:36
13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:807:68
14: tokio::runtime::context::scoped::Scoped::set
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context/scoped.rs:40:9
15: tokio::runtime::context::set_scheduler::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context.rs:180:26
16: std::thread::local::LocalKey::try_with
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:283:12
17: std::thread::local::LocalKey::with
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:260:9
18: tokio::runtime::context::set_scheduler
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context.rs:180:9
19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:807:27
20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:716:19
21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:196:28
22: tokio::runtime::context::runtime::enter_runtime
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context/runtime.rs:65:16
23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:184:9
24: tokio::runtime::runtime::Runtime::block_on_inner
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/runtime.rs:368:47
25: tokio::runtime::runtime::Runtime::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/runtime.rs:342:13
26: object_store::aws::tests::s3_test
at ./src/aws/mod.rs:610:9
27: object_store::aws::tests::s3_test::{{closure}}
at ./src/aws/mod.rs:[557](https://github.com/apache/arrow-rs/actions/runs/12672198886/job/35315696126#step:8:558):23
28: core::ops::function::FnOnce::call_once
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
29: core::ops::function::FnOnce::call_once
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
failures:
aws::tests::s3_test
test result: FAILED. 145 passed; 1 failed; 4 ignored; 0 measured; 0 filtered out; finished in 15.49s
```
**Expected behavior**
Test should pass
**Additional context**
It appears to have first failed with this PR:
- https://github.com/apache/arrow-rs/pull/6619
However, the tests passed when run on that PR ( https://github.com/apache/arrow-rs/actions/runs/11842800687/job/33002269831) which suggests to me this is related to some change in the environment / emulator rather than our code
The test passed a day before on main
- https://github.com/apache/arrow-rs/actions/runs/12640445514/job/35220876783
Contributor guide
Research direction
Start with aws::tests::s3_test in src/aws/mod.rs and the failing list_with_delimiter assertion in src/integration.rs:960. Reproduce the failure through the linked CI job and compare the emulator/environment with the referenced runs. Done means the S3 emulator test passes reliably on main.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- ci-cd, cloud, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100