awslabs / awslabs/aws-sdk-rust

Uploading this file with aws_sdk_s3::Client::put_object produces a 500

Open
#907 2 comments 0 reactions 0 assignees View on GitHub
bug p3
Dominant language
Rust
Stars
3.3k
Forks
290
Avg merge
1d 12h
Merged PRs (30d)
3

Description

### Describe the bug

I have a tool https://github.com/saethlin/crater-at-home that ends up creating pretty large HTML files and uploads them to S3. And it suddenly started failing today when I adjusted something that made this particular file not be truncated to a shorter length.

`aws s3 cp` can upload this file.

### Expected Behavior

Literally anything other than a 500. Yes this reproduces, it does not at all look like flaky behavior from the backend somewhere. I've tried a few strategies to shrink the file I'm uploading but all my ideas turned it into something that uploads successfully.

### Current Behavior

```
thread 'main' panicked at src/main.rs:19:10:
called `Result::unwrap()` on an `Err` value: ServiceError(ServiceError { source: Unhandled(Unhandled { source: ErrorMetadata { code: Some("InternalError"), message: Some("We encountered an internal error. Please try again."), extras: Some({"s3_extended_request_id":
```

### Reproduction Steps

```rust
// [dependencies]
// aws-config = "0.56.1"
// aws-sdk-s3 = "0.31.2"
// tokio = { version = "1.32.0", features = ["full"] }

#[tokio::main]
async fn main() {
let data = std::fs::read("html").unwrap();

let config = aws_config::load_from_env().await;
let client = aws_sdk_s3::Client::new(&config);
client
.put_object()
.bucket("does-aws-sdk-s3-500") // Any bucket should reproduce
.key("upload-test") // Any key should reproduce
.body(data.into())
.send()
.await
.unwrap(); // panic, response status 500
}
```
You can download the object from here: https://does-aws-sdk-s3-500.s3.amazonaws.com/upload-test

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### Version

```text
aws-sdk-bug v0.1.0 (/home/ben/aws-sdk-bug)
├── aws-config v0.56.1
│ ├── aws-credential-types v0.56.1
│ │ ├── aws-smithy-async v0.56.1
│ │ ├── aws-smithy-types v0.56.1
│ ├── aws-http v0.56.1
│ │ ├── aws-credential-types v0.56.1 (*)
│ │ ├── aws-smithy-http v0.56.1
│ │ │ ├── aws-smithy-eventstream v0.56.1
│ │ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-types v0.56.1
│ │ │ ├── aws-credential-types v0.56.1 (*)
│ │ │ ├── aws-smithy-async v0.56.1 (*)
│ │ │ ├── aws-smithy-client v0.56.1
│ │ │ │ ├── aws-smithy-async v0.56.1 (*)
│ │ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ │ ├── aws-smithy-http-tower v0.56.1
│ │ │ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ ├── aws-smithy-types v0.56.1 (*)
│ ├── aws-sdk-sso v0.30.0
│ │ ├── aws-credential-types v0.56.1 (*)
│ │ ├── aws-http v0.56.1 (*)
│ │ ├── aws-runtime v0.56.1
│ │ │ ├── aws-credential-types v0.56.1 (*)
│ │ │ ├── aws-http v0.56.1 (*)
│ │ │ ├── aws-sigv4 v0.56.1
│ │ │ │ ├── aws-smithy-eventstream v0.56.1 (*)
│ │ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ ├── aws-smithy-async v0.56.1 (*)
│ │ │ ├── aws-smithy-eventstream v0.56.1 (*)
│ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ ├── aws-smithy-runtime-api v0.56.1
│ │ │ │ ├── aws-smithy-async v0.56.1 (*)
│ │ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ │ ├── aws-types v0.56.1 (*)
│ │ ├── aws-smithy-async v0.56.1 (*)
│ │ ├── aws-smithy-client v0.56.1 (*)
│ │ ├── aws-smithy-http v0.56.1 (*)
│ │ ├── aws-smithy-json v0.56.1
│ │ │ └── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-smithy-runtime v0.56.1
│ │ │ ├── aws-smithy-async v0.56.1 (*)
│ │ │ ├── aws-smithy-client v0.56.1 (*)
│ │ │ ├── aws-smithy-http v0.56.1 (*)
│ │ │ ├── aws-smithy-runtime-api v0.56.1 (*)
│ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-smithy-runtime-api v0.56.1 (*)
│ │ ├── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-types v0.56.1 (*)
│ ├── aws-sdk-sts v0.30.0
│ │ ├── aws-credential-types v0.56.1 (*)
│ │ ├── aws-http v0.56.1 (*)
│ │ ├── aws-runtime v0.56.1 (*)
│ │ ├── aws-smithy-async v0.56.1 (*)
│ │ ├── aws-smithy-client v0.56.1 (*)
│ │ ├── aws-smithy-http v0.56.1 (*)
│ │ ├── aws-smithy-json v0.56.1 (*)
│ │ ├── aws-smithy-query v0.56.1
│ │ │ ├── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-smithy-runtime v0.56.1 (*)
│ │ ├── aws-smithy-runtime-api v0.56.1 (*)
│ │ ├── aws-smithy-types v0.56.1 (*)
│ │ ├── aws-smithy-xml v0.56.1
│ │ ├── aws-types v0.56.1 (*)
│ ├── aws-smithy-async v0.56.1 (*)
│ ├── aws-smithy-client v0.56.1 (*)
│ ├── aws-smithy-http v0.56.1 (*)
│ ├── aws-smithy-http-tower v0.56.1 (*)
│ ├── aws-smithy-json v0.56.1 (*)
│ ├── aws-smithy-types v0.56.1 (*)
│ ├── aws-types v0.56.1 (*)
├── aws-sdk-s3 v0.31.2
│ ├── aws-credential-types v0.56.1 (*)
│ ├── aws-http v0.56.1 (*)
│ ├── aws-runtime v0.56.1 (*)
│ ├── aws-sigv4 v0.56.1 (*)
│ ├── aws-smithy-async v0.56.1 (*)
│ ├── aws-smithy-checksums v0.56.1
│ │ ├── aws-smithy-http v0.56.1 (*)
│ │ ├── aws-smithy-types v0.56.1 (*)
│ ├── aws-smithy-client v0.56.1 (*)
│ ├── aws-smithy-eventstream v0.56.1 (*)
│ ├── aws-smithy-http v0.56.1 (*)
│ ├── aws-smithy-json v0.56.1 (*)
│ ├── aws-smithy-runtime v0.56.1 (*)
│ ├── aws-smithy-runtime-api v0.56.1 (*)
│ ├── aws-smithy-types v0.56.1 (*)
│ ├── aws-smithy-xml v0.56.1 (*)
│ ├── aws-types v0.56.1 (*)
```

### Environment details (OS name and version, etc.)

Arch Linux

### Logs

```
2023-10-02T01:10:08.420424Z DEBUG hyper::proto::h1::io: flushed 20266 bytes
2023-10-02T01:10:08.845658Z DEBUG hyper::proto::h1::io: parsed 7 headers
2023-10-02T01:10:08.845667Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2023-10-02T01:10:08.845717Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x11A (282 bytes)
2023-10-02T01:10:08.845737Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'before deserialization' phase
2023-10-02T01:10:08.845778Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime::client::connectors::connection_poisoning: received a transient error, poisoning the connection...
2023-10-02T01:10:08.845789Z INFO invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_http::connection: smithy connection was poisoned
2023-10-02T01:10:08.845795Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: hyper::client::connect: connection was poisoned poison_pill=PoisonPill@0x5638385423a0 { poisoned: true }
2023-10-02T01:10:08.845804Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime::client::connectors::connection_poisoning: the connection was poisoned
2023-10-02T01:10:08.845819Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'deserialization' phase
2023-10-02T01:10:08.845866Z DEBUG hyper::proto::h1::conn: incoming body completed
2023-10-02T01:10:08.845900Z DEBUG rustls::common_state: Sending warning alert CloseNotify
2023-10-02T01:10:08.846984Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt:deserialization:deserialize_nonstreaming: aws_sdk_s3::operation::put_object: extended_request_id=Some("vS1YxI9KcF5WDvpHrtfiQeGtLXTS5ZMH7fCB4Z7gr743k/pjr/q0XTmw+DIcO7mphAtkhSGLoBg=")
2023-10-02T01:10:08.846998Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt:deserialization:deserialize_nonstreaming: aws_sdk_s3::operation::put_object: request_id=Some("MSS7AJK0NXZH7KEY")
2023-10-02T01:10:08.847016Z DEBUG invoke{service=s3 operation=PutObject}:try_op:try_attempt: aws_smithy_runtime_api::client::interceptors::context: entering 'after deserialization' phase
2023-10-02T01:10:08.847030Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime::client::retries::strategy::standard: not retrying because we are out of attempts attempts=3 max_attempts=3
2023-10-02T01:10:08.847037Z DEBUG invoke{service=s3 operation=PutObject}:try_op: aws_smithy_runtime::client::orchestrator: a retry is either unnecessary or not possible, exiting attempt loop
thread 'main' panicked at src/main.rs:20:10:
```

Contributor guide

Open the contributing guide

Research direction

Start with the minimal reproduction in src/main.rs and the aws_sdk_s3::Client::put_object call, then compare its request and response logs with the successful aws s3 cp behavior. Investigate why this upload receives a 500 and determine whether the SDK can produce a successful upload for the provided HTML file; verify the reproduction no longer returns ServiceError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.