awslabs / awslabs/aws-sdk-rust

`EXEC_BAD_ACCESS` error on MacOS

Open
#1,285 2 comments 0 reactions 0 assignees View on GitHub
bug potential-regression
Dominant language
Rust
Stars
3.3k
Forks
290
Avg merge
1d 12h
Merged PRs (30d)
3

Description

### Describe the bug

I am getting some runtime errors on MacOS `EXEC_BAD_ACCESS`

I am on 1.82.0

I do not see the same issue when deployed on linux

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

### Expected Behavior

`EXEC_BAD_ACCESS` should not occur

### Current Behavior

Running on MacOS results in EXEC_BAD_ACCESS but not on linux

### Reproduction Steps

```
aws_smithy_runtime::client::interceptors::Interceptors::read_before_attempt interceptors.rs:62
aws_smithy_runtime::client::orchestrator::try_attempt::{{closure}} orchestrator.rs:88
as core::future::future::Future>::poll instrument.rs:321
aws_smithy_runtime::client::orchestrator::try_op::{{closure}}::{{closure}}::{{closure}} orchestrator.rs:313
as core::future::future::Future>::poll timeout.rs:82
aws_smithy_runtime::client::orchestrator::try_op::{{closure}}::{{closure}} orchestrator.rs:320
aws_smithy_runtime::client::orchestrator::try_op::{{closure}} orchestrator.rs:214
aws_smithy_runtime::client::orchestrator::invoke_with_stop_point::{{closure}}::{{closure}}::{{closure}} orchestrator.rs:159
as core::future::future::Future>::poll timeout.rs:82
aws_smithy_runtime::client::orchestrator::invoke_with_stop_point::{{closure}}::{{closure}} orchestrator.rs:169
as core::future::future::Future>::poll instrument.rs:321
aws_smithy_runtime::client::orchestrator::invoke_with_stop_point::{{closure}} orchestrator.rs:176
aws_sdk_s3::operation::get_object::GetObject::orchestrate_with_stop_point::{{closure}} get_object.rs:63
as core::future::future::Future>::poll instrument.rs:321
aws_sdk_s3::operation::get_object::GetObject::orchestrate::{{closure}} get_object.rs:41
aws_sdk_s3::operation::get_object::builders::GetObjectFluentBuilder::send::{{closure}} builders.rs:161
heartbeat_case::utils::s3::fetch_object::{{closure}} s3.rs:62
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::get_job_attachments_cached_no_cache::{{closure}} automated_task.rs:631
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::get_job_attachments_cached::{{closure}} automated_task.rs:608
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::AutomatedJobTaskTrait::get_job_attachments::{{closure}} automated_task.rs:516
::run_task::{{closure}} create_order.rs:104
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::AutomatedJobTaskTrait::execute_task::{{closure}} automated_task.rs:432
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::AutomatedJobTask::execute_task::{{closure}} automated_task.rs:380
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::job_tasks_utils::execute_job_tasks::{{closure}} job_tasks_utils.rs:144
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::workflow::workflow_trait::WorkflowTrait::update_job_status::{{closure}} workflow_trait.rs:144
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::workflow::workflow_trait::WorkflowTrait::complete_job::{{closure}} workflow_trait.rs:95
as core::future::future::Future>::poll future.rs:124
::complete_job::{{closure}} generic.rs:269
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::workflow::generic::GenericWorkflow::transition_job_to_initial_status::{{closure}} generic.rs:777
::create_job::{{closure}} generic.rs:624
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::workflow::workflow_trait::WorkflowTrait::create_job::{{closure}} workflow_trait.rs:212
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::job_creation_util::create_medmo_encounter_job::{{closure}} job_creation_util.rs:432
heartbeat_case::controller::handlers::job_handler::workflow::job_task::case_manager::create_follow_up_job::CreateFollowUpJobTask::create_follow_up_task::{{closure}} create_follow_up_job.rs:95
::run_task::{{closure}} create_follow_up_job.rs:44
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::AutomatedJobTaskTrait::execute_task::{{closure}} automated_task.rs:432
heartbeat_case::controller::handlers::job_handler::workflow::job_task::automated_task::AutomatedJobTask::execute_task::{{closure}} automated_task.rs:201
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::job_tasks_utils::execute_job_tasks::{{closure}} job_tasks_utils.rs:144
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::workflow::workflow_trait::WorkflowTrait::update_job_status::{{closure}} workflow_trait.rs:144
as core::future::future::Future>::poll future.rs:124
::complete_job::{{closure}} base.rs:341
as core::future::future::Future>::poll future.rs:124
::complete_job::{{closure}} generic.rs:239
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::workflow::generic::GenericWorkflow::transition_job_to_initial_status::{{closure}} generic.rs:777
::create_job::{{closure}} generic.rs:624
as core::future::future::Future>::poll future.rs:124
heartbeat_case::controller::handlers::job_handler::job::job_post_handler::{{closure}}::{{closure}} job.rs:365
as core::future::future::Future>::poll instrument.rs:321
heartbeat_case::controller::handlers::job_handler::job::job_post_handler::{{closure}} job.rs:331
>::call::{{closure}} mod.rs:239
as core::future::future::Future>::poll future.rs:124
as core::future::future::Future>::poll map.rs:55
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll macros.rs:42
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
as core::future::future::Future>::poll oneshot.rs:97
as core::future::future::Future>::poll route.rs:149
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
axum::middleware::from_fn::Next::run::{{closure}} from_fn.rs:345
heartbeat_common_lib::auth::auth_layer::combined_auth_and_service_auth::{{closure}} auth_layer.rs:209
as tower_service::Service>>::call::{{closure}} from_fn.rs:309
::poll from_fn.rs:375
::try_poll future.rs:92
as core::future::future::Future>::poll into_future.rs:34
as core::future::future::Future>::poll map.rs:55
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll macros.rs:38
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
as core::future::future::Future>::poll oneshot.rs:97
as core::future::future::Future>::poll route.rs:149
as core::future::future::Future>::poll future.rs:55
::try_poll future.rs:92
as core::future::future::Future>::poll into_future.rs:34
as core::future::future::Future>::poll map.rs:55
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll macros.rs:38
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
as core::future::future::Future>::poll oneshot.rs:97
as core::future::future::Future>::poll route.rs:149
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
axum::middleware::from_fn::Next::run::{{closure}} from_fn.rs:345
heartbeat_common_lib::utils::request_context::add_req_context::{{closure}} request_context.rs:51
as tower_service::Service>>::call::{{closure}} from_fn.rs:309
::poll from_fn.rs:375
::try_poll future.rs:92
as core::future::future::Future>::poll into_future.rs:34
as core::future::future::Future>::poll map.rs:55
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll macros.rs:38
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
as core::future::future::Future>::poll oneshot.rs:97
as core::future::future::Future>::poll route.rs:149
as core::future::future::Future>::poll request_id.rs:456
::try_poll future.rs:92
as core::future::future::Future>::poll into_future.rs:34
as core::future::future::Future>::poll map.rs:55
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll lib.rs:86
as core::future::future::Future>::poll macros.rs:38
as core::future::future::Future>::poll util.rs:94
as core::future::future::Future>::poll future.rs:124
as core::future::future::Future>::poll oneshot.rs:97
as core::future::future::Future>::poll route.rs:149
as core::future::future::Future>::poll oneshot.rs:55
as core::future::future::Future>::poll glue.rs:59
as hyper::proto::h1::dispatch::Dispatch>::poll_msg dispatch.rs:538
hyper::proto::h1::dispatch::Dispatcher::poll_write dispatch.rs:336
hyper::proto::h1::dispatch::Dispatcher::poll_loop dispatch.rs:173
hyper::proto::h1::dispatch::Dispatcher::poll_inner dispatch.rs:149
hyper::proto::h1::dispatch::Dispatcher::poll_catch dispatch.rs:128
as core::future::future::Future>::poll dispatch.rs:471
as core::future::future::Future>::poll http1.rs:530
as core::future::future::Future>::poll mod.rs:629
as core::future::future::Future>::poll future.rs:124
axum::serve::WithGracefulShutdown::run::{{closure}}::{{closure}}::{{closure}} select.rs:708
as core::future::future::Future>::poll poll_fn.rs:151
axum::serve::WithGracefulShutdown::run::{{closure}}::{{closure}} mod.rs:350
tokio::runtime::task::core::Core::poll::{{closure}} core.rs:331
[Inlined] tokio::loom::std::unsafe_cell::UnsafeCell::with_mut unsafe_cell.rs:16
tokio::runtime::task::core::Core::poll core.rs:320
tokio::runtime::task::harness::poll_future::{{closure}} harness.rs:532
as core::ops::function::FnOnce<()>>::call_once unwind_safe.rs:272
std::panicking::try::do_call panicking.rs:587
__rust_try 0x0000000102f868d4
[Inlined] std::panicking::try panicking.rs:550
std::panic::catch_unwind panic.rs:358
tokio::runtime::task::harness::poll_future harness.rs:520
tokio::runtime::task::harness::Harness::poll_inner harness.rs:209
tokio::runtime::task::harness::Harness::poll harness.rs:154
tokio::runtime::task::raw::poll raw.rs:271
tokio::runtime::task::raw::RawTask::poll raw.rs:201
tokio::runtime::task::LocalNotified::run mod.rs:463
tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}} worker.rs:674
[Inlined] tokio::task::coop::with_budget mod.rs:167
[Inlined] tokio::task::coop::budget mod.rs:133
tokio::runtime::scheduler::multi_thread::worker::Context::run_task worker.rs:594
tokio::runtime::scheduler::multi_thread::worker::Context::run worker.rs:542
tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}} worker.rs:507
tokio::runtime::context::scoped::Scoped::set scoped.rs:40
tokio::runtime::context::set_scheduler::{{closure}} context.rs:180
std::thread::local::LocalKey::try_with local.rs:310
std::thread::local::LocalKey::with local.rs:274
tokio::runtime::context::set_scheduler context.rs:180
tokio::runtime::scheduler::multi_thread::worker::run::{{closure}} worker.rs:502
tokio::runtime::context::runtime::enter_runtime runtime.rs:65
tokio::runtime::scheduler::multi_thread::worker::run worker.rs:494
tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}} worker.rs:460
as core::future::future::Future>::poll task.rs:42
tokio::runtime::task::core::Core::poll::{{closure}} core.rs:331
[Inlined] tokio::loom::std::unsafe_cell::UnsafeCell::with_mut unsafe_cell.rs:16
tokio::runtime::task::core::Core::poll core.rs:320
tokio::runtime::task::harness::poll_future::{{closure}} harness.rs:532
as core::ops::function::FnOnce<()>>::call_once unwind_safe.rs:272
std::panicking::try::do_call panicking.rs:587
__rust_try 0x000000010869f340
[Inlined] std::panicking::try panicking.rs:550
std::panic::catch_unwind panic.rs:358
tokio::runtime::task::harness::poll_future harness.rs:520
tokio::runtime::task::harness::Harness::poll_inner harness.rs:209
tokio::runtime::task::harness::Harness::poll harness.rs:154
tokio::runtime::task::raw::poll raw.rs:271
tokio::runtime::task::raw::RawTask::poll raw.rs:201
tokio::runtime::task::UnownedTask::run mod.rs:500
tokio::runtime::blocking::pool::Task::run pool.rs:161
tokio::runtime::blocking::pool::Inner::run pool.rs:511
tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}} pool.rs:469
std::sys::backtrace::__rust_begin_short_backtrace backtrace.rs:152
std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} mod.rs:559
as core::ops::function::FnOnce<()>>::call_once unwind_safe.rs:272
std::panicking::try::do_call panicking.rs:587
__rust_try 0x0000000108714b60
[Inlined] std::panicking::try panicking.rs:550
[Inlined] std::panic::catch_unwind panic.rs:358
std::thread::Builder::spawn_unchecked_::{{closure}} mod.rs:557
core::ops::function::FnOnce::call_once{{vtable.shim}} function.rs:250
[Inlined] as core::ops::function::FnOnce>::call_once boxed.rs:1976
[Inlined] as core::ops::function::FnOnce>::call_once boxed.rs:1976
std::sys::pal::unix::thread::Thread::new::thread_start thread.rs:106
_pthread_start 0x0000000180d4dc0c
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### Version

```text
1.82.0
```

### Environment details (OS name and version, etc.)

MacOS

### Logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the MacOS-only EXEC_BAD_ACCESS on version 1.82.0 and compare it with Linux. Read aws_smithy_runtime client flow at interceptors.rs:62 and orchestrator.rs:88, then follow the S3 path through s3.rs:62 and get_object.rs:63. Done means the reported S3 operation no longer crashes on MacOS and the regression is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.