apache / apache/arrow-rs-object-store

Intermittent failure with object_store aws::tests::s3_test emulator

Open
#25 2 comments 0 reactions 0 assignees View on GitHub
bug development-process
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Describe the bug**

aws::tests::s3_test emulator CI tests are failing on main

**To Reproduce**

https://github.com/apache/arrow-rs/actions/runs/12672198886/job/35315696126

```
---- aws::tests::s3_test stdout ----
thread 'aws::tests::s3_test' panicked at src/integration.rs:960:5:
assertion failed: content_list.is_empty()
stack backtrace:
0: rust_begin_unwind
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5
3: object_store::integration::list_with_delimiter::{{closure}}
at ./src/integration.rs:960:5
4: object_store::aws::tests::s3_test::{{closure}}
at ./src/aws/mod.rs:569:43
5: as core::future::future::Future>::poll
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/future/future.rs:123:9
6: as core::future::future::Future>::poll
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/future/future.rs:123:9
7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:729:57
8: tokio::runtime::coop::with_budget
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/coop.rs:107:5
9: tokio::runtime::coop::budget
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/coop.rs:73:5
10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:729:25
11: tokio::runtime::scheduler::current_thread::Context::enter
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:428:19
12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:728:36
13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:807:68
14: tokio::runtime::context::scoped::Scoped::set
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context/scoped.rs:40:9
15: tokio::runtime::context::set_scheduler::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context.rs:180:26
16: std::thread::local::LocalKey::try_with
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:283:12
17: std::thread::local::LocalKey::with
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:260:9
18: tokio::runtime::context::set_scheduler
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context.rs:180:9
19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:807:27
20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:716:19
21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:196:28
22: tokio::runtime::context::runtime::enter_runtime
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/context/runtime.rs:65:16
23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/scheduler/current_thread/mod.rs:184:9
24: tokio::runtime::runtime::Runtime::block_on_inner
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/runtime.rs:368:47
25: tokio::runtime::runtime::Runtime::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.42.0/src/runtime/runtime.rs:342:13
26: object_store::aws::tests::s3_test
at ./src/aws/mod.rs:610:9
27: object_store::aws::tests::s3_test::{{closure}}
at ./src/aws/mod.rs:[557](https://github.com/apache/arrow-rs/actions/runs/12672198886/job/35315696126#step:8:558):23
28: core::ops::function::FnOnce::call_once
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
29: core::ops::function::FnOnce::call_once
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

failures:
aws::tests::s3_test

test result: FAILED. 145 passed; 1 failed; 4 ignored; 0 measured; 0 filtered out; finished in 15.49s
```
**Expected behavior**
Test should pass

**Additional context**
It appears to have first failed with this PR:
- https://github.com/apache/arrow-rs/pull/6619

However, the tests passed when run on that PR ( https://github.com/apache/arrow-rs/actions/runs/11842800687/job/33002269831) which suggests to me this is related to some change in the environment / emulator rather than our code

The test passed a day before on main
- https://github.com/apache/arrow-rs/actions/runs/12640445514/job/35220876783

Contributor guide

Open the contributing guide

Research direction

Start with aws::tests::s3_test in src/aws/mod.rs and the failing list_with_delimiter assertion in src/integration.rs:960. Reproduce the failure through the linked CI job and compare the emulator/environment with the referenced runs. Done means the S3 emulator test passes reliably on main.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
ci-cd, cloud, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.