aws / aws/aws-sdk-cpp

SIGSEGV in S3CrtRequestHeadersCallback — use-after-free reading ContinueRequestHandler from destroyed request

Open
#3,881 2 comments 0 reactions 0 assignees View on GitHub
bug feature-request
Dominant language
C++
Stars
2.2k
Forks
1.2k
Avg merge
3d 14h
Merged PRs (30d)
12

Description

### Describe the bug

`S3CrtClient` stores a **raw pointer** to the caller's request
(`CrtRequestCallbackUserData::originalRequest = &request`, `S3CrtClient.cpp`
`InitCommonCrtRequestOption`), which leads to a segfault when passing in a stack-local request.

The crash was introduced by upstream [PR #3231 "Cancel S3 CRT requests if should not continue"](https://github.com/aws/aws-sdk-cpp/pull/3231). The dangling `originalRequest` raw pointer existed before that but was not dereferenced.

This is one of six locations where the dangling pointer is dereferenced, the others may be subject to a similar failure, though I did not observe that. This behavior is different in `S3Client`, which captures the passed request by value, avoiding this failure family.

### Regression Issue

- [x] Select this option if this issue appears to be a regression.

### Expected Behavior

The program should not crash with a segfault.

### Current Behavior

The `*Async` submit call returns as soon as the request is queued with the CRT, so
a caller that does not keep its request object alive until completion (a common and
previously-safe pattern) has its request destroyed **before** the response arrives.
When the HTTP response headers are processed, the callback reads
`GetContinueRequestHandler()` out of the freed request and invokes the resulting
dangling `std::function`, crashing with `SIGSEGV`:

```
#0 0x0 in ?? ()
#1 Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (response_status=200) at S3CrtClient.cpp:577
#2 s_s3_meta_request_default_request_finished at aws-c-s3/source/s3_default_meta_request.c
#3 aws_s3_meta_request_finished_request at aws-c-s3/source/s3_meta_request.c
...
aws_event_loop_thread at aws-c-io/source/linux/epoll_event_loop.c
```

### Reproduction Steps

Using a small example:
```cpp
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include

static std::shared_ptr client;
static std::atomic done{0};

// The request is a LOCAL: it is destroyed the moment this function returns, long
// before the async HTTP 200 response is processed on an AwsEventLoop thread.
static void submit(uint8_t* buf, size_t len) {
Aws::S3Crt::Model::PutObjectRequest req;
req.SetBucket("bck");
req.SetKey("1/1");
req.SetContinueRequestHandler([](const Aws::Http::HttpRequest*) { return true; });
req.SetBody(Aws::MakeShared(
"", new Aws::Utils::Stream::PreallocatedStreamBuf(buf, len)));
client->PutObjectAsync(req,
[](const Aws::S3Crt::S3CrtClient*, const Aws::S3Crt::Model::PutObjectRequest&,
const Aws::S3Crt::Model::PutObjectOutcome&,
const std::shared_ptr&) {
done.fetch_add(1);
}, nullptr);
// <-- `req` destroyed here; userData->originalRequest now dangles.
}

int main() {
const char* ep = std::getenv("MINIO_ENDPOINT");
if (!ep) { std::puts("set MINIO_ENDPOINT"); return 0; }
Aws::SDKOptions o; Aws::InitAPI(o);
{
Aws::S3Crt::S3CrtClientConfiguration cfg;
cfg.scheme = Aws::Http::Scheme::HTTP;
cfg.verifySSL = false;
cfg.region = "us-east-1";
cfg.useVirtualAddressing = false; // path-style, for MinIO
cfg.endpointOverride = ep;
Aws::Auth::AWSCredentials creds(std::getenv("AWS_ACCESS_KEY_ID"),
std::getenv("AWS_SECRET_ACCESS_KEY"));
client = Aws::MakeShared(
"repro", creds, cfg,
Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false);

Aws::S3Crt::Model::CreateBucketRequest cb; cb.SetBucket("bck");
client->CreateBucket(cb); // ignore "already owned"

static uint8_t payload[4096];
std::memset(payload, 0xAB, sizeof(payload));
for (int i = 0; i < 16; i++) submit(payload, sizeof(payload)); // crashes here (pre-fix)

for (int s = 0; s < 500 && done.load() < 16; s++)
std::this_thread::sleep_for(std::chrono::milliseconds(10));
std::printf("completed %d/16\n", done.load());
client.reset();
}
Aws::ShutdownAPI(o);
return 0;
}
```
reliably crashes:
```
$ docker run -d -p 9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin quay.io/minio/minio server /data
$ g++ -std=c++17 -g test.cpp -o repro -laws-cpp-sdk-s3-crt -laws-cpp-sdk-core -pthread
$ MINIO_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin ./repro
Segmentation fault (core dumped) MINIO_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin ./repro
$ MINIO_ENDPOINT=http://127.0.0.1:9000 AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin gdb ./repro
GNU gdb (GDB) 17.2
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test2...
(gdb) r
...
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff62ba6c0 (LWP 8981)]
[New Thread 0x7ffff5ab96c0 (LWP 8982)]
[New Thread 0x7ffff52b86c0 (LWP 8983)]
[New Thread 0x7ffff4ab76c0 (LWP 8984)]
[New Thread 0x7fffe7fff6c0 (LWP 8985)]
[New Thread 0x7fffe77fe6c0 (LWP 8986)]
[New Thread 0x7fffe6ffd6c0 (LWP 8995)]
[New Thread 0x7fffe67fc6c0 (LWP 8996)]
[New Thread 0x7fffe5ffb6c0 (LWP 8997)]
[New Thread 0x7fffe57fa6c0 (LWP 8998)]
[New Thread 0x7fffe4ff96c0 (LWP 8999)]
[New Thread 0x7fffc3fff6c0 (LWP 9000)]
[New Thread 0x7fffc37fe6c0 (LWP 9001)]

Thread 6 "AwsEventLoop5" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe7fff6c0 (LWP 8985)]
thread 6

Downloading 201.34 K source file /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp
0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=, headers=0x7fffdc014b70, response_status=, user_data=)
at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
573 auto& shouldContinueFn = userData->originalRequest->GetContinueRequestHandler();
(gdb) thread 6
[Switching to thread 6 (Thread 0x7fffe7fff6c0 (LWP 8985))]
#0 0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=, headers=0x7fffdc014b70, response_status=, user_data=)
at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
573 auto& shouldContinueFn = userData->originalRequest->GetContinueRequestHandler();
(gdb)
[Switching to thread 6 (Thread 0x7fffe7fff6c0 (LWP 8985))]
#0 0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=, headers=0x7fffdc014b70, response_status=, user_data=)
at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
573 auto& shouldContinueFn = userData->originalRequest->GetContinueRequestHandler();
(gdb) bt
#0 0x00007ffff7d545b7 in Aws::S3Crt::S3CrtClient::S3CrtRequestHeadersCallback (meta_request=, headers=0x7fffdc014b70, response_status=, user_data=)
at /usr/src/debug/aws-sdk-cpp/aws-sdk-cpp-1.11.792/generated/src/aws-cpp-sdk-s3-crt/source/S3CrtClient.cpp:573
#1 0x00007ffff797f39f in s_s3_meta_request_default_request_finished (meta_request=0x555555924610, request=0x7fffe8013f60, error_code=) at /usr/src/debug/aws-c-s3/aws-c-s3-0.12.2/source/s3_default_meta_request.c:417
#2 0x00007ffff79791b2 in aws_s3_client_notify_connection_finished (client=0x5555559789c0, connection=0x7fffe8014770, error_code=0, finish_code=) at /usr/src/debug/aws-c-s3/aws-c-s3-0.12.2/source/s3_client.c:2640
#3 0x00007ffff75b0864 in s_stream_complete (stream=0x7fffdc0120e0, error_code=) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:741
#4 0x00007ffff75b2240 in s_decoder_on_done (user_data=0x7fffdc011ba0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1504
#5 0x00007ffff75b7c3a in s_mark_done (decoder=0x7fffdc011e60) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:199
#6 s_linestate_header (decoder=0x7fffdc011e60, input=...) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:372
#7 0x00007ffff75b3da5 in s_state_getline (decoder=0x7fffdc011e60, input=0x7fffe7ffd2a0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:128
#8 0x00007ffff75b5cc5 in aws_h1_decode (decoder=0x7fffdc011e60, data=data@entry=0x7fffe7ffd2a0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_decoder.c:742
#9 0x00007ffff75b7116 in s_try_process_next_stream_read_message (connection=0x7fffdc011ba0, out_stop_processing=) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:2057
#10 aws_h1_connection_try_process_read_messages (connection=0x7fffdc011ba0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1923
#11 0x00007ffff75acb89 in aws_h1_connection_try_process_read_messages (connection=0x7fffdc011ba0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1896
#12 s_handler_process_read_message (handler=, slot=, message=0x7fffdc00d5e0) at /usr/src/debug/aws-c-http/aws-c-http-0.10.14/source/h1_connection.c:1889
#13 0x00007ffff75790e6 in s_do_read (socket_handler=0x7fffe8014bd0) at /usr/src/debug/aws-c-io/aws-c-io-0.26.3/source/socket_channel_handler.c:166
#14 0x00007ffff7577853 in s_on_socket_io_event (event_loop=, handle=, events=3, user_data=0x7fffdc000c10) at /usr/src/debug/aws-c-io/aws-c-io-0.26.3/source/posix/socket.c:1880
#15 0x00007ffff756ef1f in aws_event_loop_thread (args=) at /usr/src/debug/aws-c-io/aws-c-io-0.26.3/source/linux/epoll_event_loop.c:686
#16 0x00007ffff790523a in thread_fn (arg=0x555555682f60) at /usr/src/debug/aws-c-common/aws-c-common-0.12.6/source/posix/thread.c:177
#17 0x00007ffff7297739 in ?? () from /usr/lib/libc.so.6
#18 0x00007ffff731bedc in ?? () from /usr/lib/libc.so.6
```

### Possible Solution

A hotfix that copies the `ContinueRequestHandler` into the `CrtCallbackRequestUserData` works around this issue. The root cause, the dangling `originalRequest` pointer remains though, potentially leading to other issues when dereferencing it. A proper solution should copy the request into the user data, similar to how `S3Client` captures it by value.

### Additional Information/Context

_No response_

### AWS CPP SDK version used

1.11.792 and 1.11.855

### Compiler and Version used

gcc 16.1.1

### Operating System and version

Archlinux

Contributor guide

Open the contributing guide

Research direction

Start in S3CrtClient.cpp at InitCommonCrtRequestOption and S3CrtRequestHeadersCallback, then inspect the other five dereferences of originalRequest. Use the provided stack-local PutObjectRequest reproduction with MinIO to verify the failure and confirm that asynchronous requests remain safe after the caller returns without a SIGSEGV.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.